Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 14:08:50.267822 24317 test_util.cc:268] Using random seed: 110868113
I20240501 14:08:50.274251 24317 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:08:50.274315 24317 ts_itest-base.cc:113] --------------
I20240501 14:08:50.274333 24317 ts_itest-base.cc:114] 6 tablet servers
I20240501 14:08:50.274348 24317 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:08:50.274360 24317 ts_itest-base.cc:116] --------------
2024-05-01T14:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:08:50Z Disabled control of system clock
I20240501 14:08:50.283233 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.191.126:46765
--webserver_interface=127.23.191.126
--webserver_port=0
--builtin_ntp_servers=127.23.191.84:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.191.126:46765
--location_mapping_cmd=/tmp/dist-test-tasklfWEhd/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 14:08:50.384882 24329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:50.385092 24329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:50.385128 24329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:50.388067 24329 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:50.388129 24329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:08:50.388159 24329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:50.388175 24329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:08:50.388197 24329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:08:50.391712 24329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-tasklfWEhd/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.23.191.126:46765
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.191.126:46765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.23.191.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.24329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:50.392517 24329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:50.393574 24329 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:50.400339 24334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.400490 24337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.400774 24335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:50.400996 24329 server_base.cc:1034] running on GCE node
I20240501 14:08:50.401463 24329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:50.402310 24329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:50.403451 24329 hybrid_clock.cc:648] HybridClock initialized: now 1714572530403435 us; error 35 us; skew 500 ppm
I20240501 14:08:50.405270 24329 webserver.cc:460] Webserver started at http://127.23.191.126:43157/ using document root <none> and password file <none>
I20240501 14:08:50.405793 24329 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:50.405853 24329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:50.406014 24329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:50.407600 24329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5f54f40816644fd3bef271c813eec707"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-c4g3"
I20240501 14:08:50.408046 24329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5f54f40816644fd3bef271c813eec707"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-c4g3"
I20240501 14:08:50.411209 24329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20240501 14:08:50.413277 24343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.414247 24329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:08:50.414366 24329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5f54f40816644fd3bef271c813eec707"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-c4g3"
I20240501 14:08:50.414456 24329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:50.424659 24329 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:50.425267 24329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:50.425424 24329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:50.431291 24329 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.126:46765
I20240501 14:08:50.431295 24395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.126:46765 every 8 connection(s)
I20240501 14:08:50.432200 24329 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:08:50.434652 24396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:50.438231 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 24329
I20240501 14:08:50.438396 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:08:50.440210 24396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707: Bootstrap starting.
I20240501 14:08:50.442747 24396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:50.443717 24396 log.cc:824] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:50.445711 24396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707: No bootstrap required, opened a new log
I20240501 14:08:50.448627 24396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f54f40816644fd3bef271c813eec707" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 46765 } }
I20240501 14:08:50.448854 24396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:50.448921 24396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f54f40816644fd3bef271c813eec707, State: Initialized, Role: FOLLOWER
I20240501 14:08:50.449393 24396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f54f40816644fd3bef271c813eec707" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 46765 } }
I20240501 14:08:50.449548 24396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:50.449620 24396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:50.449728 24396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.450731 24396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f54f40816644fd3bef271c813eec707" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 46765 } }
I20240501 14:08:50.451103 24396 leader_election.cc:302] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f54f40816644fd3bef271c813eec707; no voters:
I20240501 14:08:50.451418 24396 leader_election.cc:288] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:50.451530 24401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:50.452056 24401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [term 1 LEADER]: Becoming Leader. State: Replica: 5f54f40816644fd3bef271c813eec707, State: Running, Role: LEADER
I20240501 14:08:50.452415 24401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f54f40816644fd3bef271c813eec707" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 46765 } }
I20240501 14:08:50.452670 24396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:50.453975 24403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f54f40816644fd3bef271c813eec707. Latest consensus state: current_term: 1 leader_uuid: "5f54f40816644fd3bef271c813eec707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f54f40816644fd3bef271c813eec707" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 46765 } } }
I20240501 14:08:50.454097 24403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:50.454402 24402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f54f40816644fd3bef271c813eec707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f54f40816644fd3bef271c813eec707" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 46765 } } }
I20240501 14:08:50.454493 24402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707 [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:50.454886 24410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:08:50.458186 24410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:08:50.463285 24410 catalog_manager.cc:1347] Generated new cluster ID: 4a3a619f993445bcbd91e8f11fd3aadb
I20240501 14:08:50.463346 24410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:08:50.479431 24410 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:08:50.480329 24410 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:08:50.485831 24410 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5f54f40816644fd3bef271c813eec707: Generated new TSK 0
I20240501 14:08:50.486498 24410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:08:50.501298 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.65:0
--local_ip_for_outbound_sockets=127.23.191.65
--webserver_interface=127.23.191.65
--webserver_port=0
--tserver_master_addrs=127.23.191.126:46765
--builtin_ntp_servers=127.23.191.84:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:08:50.606036 24420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:50.606246 24420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:50.606309 24420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:50.609337 24420 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:50.609405 24420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:50.609486 24420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.65
I20240501 14:08:50.613058 24420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.23.191.65
--webserver_port=0
--tserver_master_addrs=127.23.191.126:46765
--never_fsync=true
--heap_profile_path=/tmp/kudu.24420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.65
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:50.613909 24420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:50.615010 24420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:50.622730 24425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.622943 24428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.623123 24426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:50.623375 24420 server_base.cc:1034] running on GCE node
I20240501 14:08:50.623733 24420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:50.624248 24420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:50.625388 24420 hybrid_clock.cc:648] HybridClock initialized: now 1714572530625372 us; error 30 us; skew 500 ppm
I20240501 14:08:50.631537 24420 webserver.cc:460] Webserver started at http://127.23.191.65:33265/ using document root <none> and password file <none>
I20240501 14:08:50.632076 24420 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:50.632138 24420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:50.632306 24420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:50.633956 24420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-c4g3"
I20240501 14:08:50.634425 24420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-c4g3"
I20240501 14:08:50.637570 24420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:08:50.639576 24434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.640516 24420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:50.640619 24420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-c4g3"
I20240501 14:08:50.640697 24420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:50.662572 24420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:50.663239 24420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:50.663404 24420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:50.663940 24420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:50.664853 24420 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:50.664908 24420 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.664963 24420 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:50.664986 24420 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.673017 24420 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.65:43561
I20240501 14:08:50.673071 24547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.65:43561 every 8 connection(s)
I20240501 14:08:50.673964 24420 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:08:50.677042 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 24420
I20240501 14:08:50.677134 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:08:50.679114 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.66:0
--local_ip_for_outbound_sockets=127.23.191.66
--webserver_interface=127.23.191.66
--webserver_port=0
--tserver_master_addrs=127.23.191.126:46765
--builtin_ntp_servers=127.23.191.84:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:08:50.683027 24548 heartbeater.cc:347] Connected to a master server at 127.23.191.126:46765
I20240501 14:08:50.683246 24548 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:50.683751 24548 heartbeater.cc:511] Master 127.23.191.126:46765 requested a full tablet report, sending...
I20240501 14:08:50.723558 24360 ts_manager.cc:194] Registered new tserver with Master: fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:08:50.725379 24360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.65:49323
W20240501 14:08:50.786985 24551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:50.787194 24551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:50.787232 24551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:50.790195 24551 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:50.790253 24551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:50.790315 24551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.66
I20240501 14:08:50.793813 24551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.23.191.66
--webserver_port=0
--tserver_master_addrs=127.23.191.126:46765
--never_fsync=true
--heap_profile_path=/tmp/kudu.24551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.66
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:50.794677 24551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:50.795766 24551 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:50.803588 24557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.803653 24558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.803992 24560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:50.804315 24551 server_base.cc:1034] running on GCE node
I20240501 14:08:50.804665 24551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:50.805254 24551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:50.806404 24551 hybrid_clock.cc:648] HybridClock initialized: now 1714572530806386 us; error 45 us; skew 500 ppm
I20240501 14:08:50.808076 24551 webserver.cc:460] Webserver started at http://127.23.191.66:43277/ using document root <none> and password file <none>
I20240501 14:08:50.808615 24551 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:50.808676 24551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:50.808879 24551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:50.810521 24551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f68136de04c94e5cb4a5f99881f02ab3"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-c4g3"
I20240501 14:08:50.810967 24551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f68136de04c94e5cb4a5f99881f02ab3"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-c4g3"
I20240501 14:08:50.814085 24551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:08:50.816094 24566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.817080 24551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:50.817178 24551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f68136de04c94e5cb4a5f99881f02ab3"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-c4g3"
I20240501 14:08:50.817255 24551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:50.834350 24551 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:50.834932 24551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:50.835059 24551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:50.835522 24551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:50.836336 24551 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:50.836380 24551 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.836416 24551 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:50.836436 24551 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.844313 24551 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.66:40345
I20240501 14:08:50.844341 24679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.66:40345 every 8 connection(s)
I20240501 14:08:50.845244 24551 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:08:50.854264 24680 heartbeater.cc:347] Connected to a master server at 127.23.191.126:46765
I20240501 14:08:50.854470 24680 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:50.854894 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 24551
I20240501 14:08:50.854981 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:08:50.855021 24680 heartbeater.cc:511] Master 127.23.191.126:46765 requested a full tablet report, sending...
I20240501 14:08:50.858839 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.67:0
--local_ip_for_outbound_sockets=127.23.191.67
--webserver_interface=127.23.191.67
--webserver_port=0
--tserver_master_addrs=127.23.191.126:46765
--builtin_ntp_servers=127.23.191.84:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:08:50.887419 24360 ts_manager.cc:194] Registered new tserver with Master: f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:08:50.888120 24360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.66:51321
W20240501 14:08:50.963477 24684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:50.963682 24684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:50.963728 24684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:50.966722 24684 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:50.966781 24684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:50.966877 24684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.67
I20240501 14:08:50.970404 24684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.23.191.67
--webserver_port=0
--tserver_master_addrs=127.23.191.126:46765
--never_fsync=true
--heap_profile_path=/tmp/kudu.24684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.67
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:50.971277 24684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:50.972424 24684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:50.980270 24690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.980499 24689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.980654 24692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:50.980989 24684 server_base.cc:1034] running on GCE node
I20240501 14:08:50.981364 24684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:50.981896 24684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:50.983047 24684 hybrid_clock.cc:648] HybridClock initialized: now 1714572530983033 us; error 26 us; skew 500 ppm
I20240501 14:08:50.984923 24684 webserver.cc:460] Webserver started at http://127.23.191.67:46711/ using document root <none> and password file <none>
I20240501 14:08:50.985464 24684 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:50.985515 24684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:50.985680 24684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:50.987334 24684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-c4g3"
I20240501 14:08:50.987800 24684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-c4g3"
I20240501 14:08:50.991052 24684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:08:50.993288 24698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.994302 24684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:50.994426 24684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-c4g3"
I20240501 14:08:50.994524 24684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:51.016824 24684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:51.017529 24684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:51.017683 24684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:51.018271 24684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:51.019201 24684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:51.019255 24684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.019320 24684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:51.019347 24684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.027626 24684 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.67:40311
I20240501 14:08:51.027649 24811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.67:40311 every 8 connection(s)
I20240501 14:08:51.028522 24684 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:08:51.033999 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 24684
I20240501 14:08:51.034081 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:08:51.036286 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.68:0
--local_ip_for_outbound_sockets=127.23.191.68
--webserver_interface=127.23.191.68
--webserver_port=0
--tserver_master_addrs=127.23.191.126:46765
--builtin_ntp_servers=127.23.191.84:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:08:51.037191 24812 heartbeater.cc:347] Connected to a master server at 127.23.191.126:46765
I20240501 14:08:51.037408 24812 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:51.037920 24812 heartbeater.cc:511] Master 127.23.191.126:46765 requested a full tablet report, sending...
I20240501 14:08:51.076179 24360 ts_manager.cc:194] Registered new tserver with Master: 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:08:51.077203 24360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.67:54567
W20240501 14:08:51.144610 24815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:51.144853 24815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:51.144906 24815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:51.147900 24815 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:51.147964 24815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:51.148032 24815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.68
I20240501 14:08:51.151610 24815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.23.191.68
--webserver_port=0
--tserver_master_addrs=127.23.191.126:46765
--never_fsync=true
--heap_profile_path=/tmp/kudu.24815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.68
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:51.152470 24815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:51.153679 24815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:51.161417 24821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:51.161402 24822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:51.161698 24824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:51.161964 24815 server_base.cc:1034] running on GCE node
I20240501 14:08:51.162319 24815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:51.162849 24815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:51.163993 24815 hybrid_clock.cc:648] HybridClock initialized: now 1714572531163977 us; error 28 us; skew 500 ppm
I20240501 14:08:51.165803 24815 webserver.cc:460] Webserver started at http://127.23.191.68:43109/ using document root <none> and password file <none>
I20240501 14:08:51.166335 24815 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:51.166394 24815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:51.166558 24815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:51.168162 24815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c2691129fd82431bb867cb1307e009b6"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-c4g3"
I20240501 14:08:51.168623 24815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c2691129fd82431bb867cb1307e009b6"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-c4g3"
I20240501 14:08:51.171859 24815 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:51.173976 24830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.174960 24815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:51.175060 24815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c2691129fd82431bb867cb1307e009b6"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-c4g3"
I20240501 14:08:51.175139 24815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:51.203109 24815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:51.203800 24815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:51.203956 24815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:51.204515 24815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:51.205444 24815 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:51.205493 24815 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.205547 24815 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:51.205574 24815 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.213680 24815 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.68:44971
I20240501 14:08:51.213711 24943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.68:44971 every 8 connection(s)
I20240501 14:08:51.214601 24815 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:08:51.221954 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 24815
I20240501 14:08:51.222055 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:08:51.223280 24944 heartbeater.cc:347] Connected to a master server at 127.23.191.126:46765
I20240501 14:08:51.223503 24944 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:51.224009 24944 heartbeater.cc:511] Master 127.23.191.126:46765 requested a full tablet report, sending...
I20240501 14:08:51.224071 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.69:0
--local_ip_for_outbound_sockets=127.23.191.69
--webserver_interface=127.23.191.69
--webserver_port=0
--tserver_master_addrs=127.23.191.126:46765
--builtin_ntp_servers=127.23.191.84:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:08:51.255470 24359 ts_manager.cc:194] Registered new tserver with Master: c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971)
I20240501 14:08:51.256191 24359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.68:49525
W20240501 14:08:51.329003 24947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:51.329224 24947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:51.329284 24947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:51.332286 24947 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:51.332356 24947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:51.332445 24947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.69
I20240501 14:08:51.336007 24947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.23.191.69
--webserver_port=0
--tserver_master_addrs=127.23.191.126:46765
--never_fsync=true
--heap_profile_path=/tmp/kudu.24947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.69
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:51.336922 24947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:51.338047 24947 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:51.345902 24956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:51.346109 24953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:51.346294 24954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:51.346542 24947 server_base.cc:1034] running on GCE node
I20240501 14:08:51.346933 24947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:51.347466 24947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:51.348615 24947 hybrid_clock.cc:648] HybridClock initialized: now 1714572531348602 us; error 30 us; skew 500 ppm
I20240501 14:08:51.350461 24947 webserver.cc:460] Webserver started at http://127.23.191.69:42347/ using document root <none> and password file <none>
I20240501 14:08:51.350994 24947 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:51.351051 24947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:51.351215 24947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:51.352903 24947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7e375044d63649c097759c4b667eff79"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-c4g3"
I20240501 14:08:51.353379 24947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7e375044d63649c097759c4b667eff79"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-c4g3"
I20240501 14:08:51.356668 24947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 14:08:51.358741 24962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.359732 24947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:51.359830 24947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7e375044d63649c097759c4b667eff79"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-c4g3"
I20240501 14:08:51.359916 24947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:51.377250 24947 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:51.377875 24947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:51.378028 24947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:51.378621 24947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:51.379521 24947 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:51.379575 24947 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.379617 24947 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:51.379645 24947 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.387938 24947 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.69:45635
I20240501 14:08:51.387981 25075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.69:45635 every 8 connection(s)
I20240501 14:08:51.388885 24947 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:08:51.389297 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 24947
I20240501 14:08:51.389418 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:08:51.391695 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.70:0
--local_ip_for_outbound_sockets=127.23.191.70
--webserver_interface=127.23.191.70
--webserver_port=0
--tserver_master_addrs=127.23.191.126:46765
--builtin_ntp_servers=127.23.191.84:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:08:51.398605 25076 heartbeater.cc:347] Connected to a master server at 127.23.191.126:46765
I20240501 14:08:51.398833 25076 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:51.399382 25076 heartbeater.cc:511] Master 127.23.191.126:46765 requested a full tablet report, sending...
I20240501 14:08:51.433954 24359 ts_manager.cc:194] Registered new tserver with Master: 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635)
I20240501 14:08:51.434851 24359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.69:39941
W20240501 14:08:51.500494 25079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:51.500702 25079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:51.500746 25079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:51.503768 25079 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:51.503831 25079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:51.503931 25079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.70
I20240501 14:08:51.507524 25079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.23.191.70
--webserver_port=0
--tserver_master_addrs=127.23.191.126:46765
--never_fsync=true
--heap_profile_path=/tmp/kudu.25079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.70
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:51.508354 25079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:51.509467 25079 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:51.517071 25085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:51.517120 25086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:51.517347 25088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:51.517632 25079 server_base.cc:1034] running on GCE node
I20240501 14:08:51.518007 25079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:51.518607 25079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:51.519748 25079 hybrid_clock.cc:648] HybridClock initialized: now 1714572531519730 us; error 37 us; skew 500 ppm
I20240501 14:08:51.521646 25079 webserver.cc:460] Webserver started at http://127.23.191.70:33773/ using document root <none> and password file <none>
I20240501 14:08:51.522167 25079 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:51.522221 25079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:51.522410 25079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:51.524039 25079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "786f81b7c57f452e9b463f721fc3e865"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-c4g3"
I20240501 14:08:51.524489 25079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "786f81b7c57f452e9b463f721fc3e865"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-c4g3"
I20240501 14:08:51.527637 25079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:08:51.529695 25094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.530634 25079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:51.530743 25079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "786f81b7c57f452e9b463f721fc3e865"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-c4g3"
I20240501 14:08:51.530822 25079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:51.568259 25079 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:51.568926 25079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:51.569079 25079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:51.569810 25079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:51.570756 25079 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:51.570804 25079 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.570855 25079 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:51.570883 25079 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.578963 25079 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.70:35343
I20240501 14:08:51.579008 25208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.70:35343 every 8 connection(s)
I20240501 14:08:51.579839 25079 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:08:51.587544 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 25079
I20240501 14:08:51.587641 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:08:51.587939 25209 heartbeater.cc:347] Connected to a master server at 127.23.191.126:46765
I20240501 14:08:51.588167 25209 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:51.588670 25209 heartbeater.cc:511] Master 127.23.191.126:46765 requested a full tablet report, sending...
I20240501 14:08:51.621313 24360 ts_manager.cc:194] Registered new tserver with Master: 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343)
I20240501 14:08:51.622047 24360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.70:44295
I20240501 14:08:51.625507 24317 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 14:08:51.636149 24317 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 14:08:51.727984 24548 heartbeater.cc:503] Master 127.23.191.126:46765 was elected leader, sending a full tablet report...
I20240501 14:08:51.890347 24680 heartbeater.cc:503] Master 127.23.191.126:46765 was elected leader, sending a full tablet report...
I20240501 14:08:52.079782 24812 heartbeater.cc:503] Master 127.23.191.126:46765 was elected leader, sending a full tablet report...
I20240501 14:08:52.258347 24944 heartbeater.cc:503] Master 127.23.191.126:46765 was elected leader, sending a full tablet report...
I20240501 14:08:52.437495 25076 heartbeater.cc:503] Master 127.23.191.126:46765 was elected leader, sending a full tablet report...
I20240501 14:08:52.624192 25209 heartbeater.cc:503] Master 127.23.191.126:46765 was elected leader, sending a full tablet report...
I20240501 14:08:55.388574 24360 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43068:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:55.410521 24614 tablet_service.cc:1467] Processing CreateTablet for tablet a5fd0fa373d74cd184b6626c559e3ee0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7141caae5ac544368ba776ded74293cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.414594 24614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5fd0fa373d74cd184b6626c559e3ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.414352 24482 tablet_service.cc:1467] Processing CreateTablet for tablet 304de776104a4ab382e9274e3006be59 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7141caae5ac544368ba776ded74293cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.414758 25143 tablet_service.cc:1467] Processing CreateTablet for tablet 304de776104a4ab382e9274e3006be59 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7141caae5ac544368ba776ded74293cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.415340 24482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304de776104a4ab382e9274e3006be59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.415601 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304de776104a4ab382e9274e3006be59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.415505 24746 tablet_service.cc:1467] Processing CreateTablet for tablet 304de776104a4ab382e9274e3006be59 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7141caae5ac544368ba776ded74293cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.416306 24746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304de776104a4ab382e9274e3006be59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.418959 25010 tablet_service.cc:1467] Processing CreateTablet for tablet a5fd0fa373d74cd184b6626c559e3ee0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7141caae5ac544368ba776ded74293cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.419859 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5fd0fa373d74cd184b6626c559e3ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.415508 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 29231eab3ff84873a05a65ad6513c4e5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7141caae5ac544368ba776ded74293cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.420415 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29231eab3ff84873a05a65ad6513c4e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.421175 24481 tablet_service.cc:1467] Processing CreateTablet for tablet 29231eab3ff84873a05a65ad6513c4e5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7141caae5ac544368ba776ded74293cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.421451 24481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29231eab3ff84873a05a65ad6513c4e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.422070 25009 tablet_service.cc:1467] Processing CreateTablet for tablet 29231eab3ff84873a05a65ad6513c4e5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7141caae5ac544368ba776ded74293cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.423036 25236 tablet_bootstrap.cc:492] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:08:55.424067 25235 tablet_bootstrap.cc:492] T 304de776104a4ab382e9274e3006be59 P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:08:55.425330 25236 tablet_bootstrap.cc:654] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.425334 24878 tablet_service.cc:1467] Processing CreateTablet for tablet a5fd0fa373d74cd184b6626c559e3ee0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7141caae5ac544368ba776ded74293cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.426110 25237 tablet_bootstrap.cc:492] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:08:55.426148 25235 tablet_bootstrap.cc:654] T 304de776104a4ab382e9274e3006be59 P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.426159 25236 log.cc:824] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:55.426276 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5fd0fa373d74cd184b6626c559e3ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.427153 25235 log.cc:824] T 304de776104a4ab382e9274e3006be59 P 319fbe37b2ce4434a8d2fe879ca920a7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:55.427982 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29231eab3ff84873a05a65ad6513c4e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.428081 25237 tablet_bootstrap.cc:654] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.429307 25237 log.cc:824] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:55.430104 25236 tablet_bootstrap.cc:492] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:08:55.430127 25235 tablet_bootstrap.cc:492] T 304de776104a4ab382e9274e3006be59 P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:08:55.430279 25235 ts_tablet_manager.cc:1367] T 304de776104a4ab382e9274e3006be59 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 14:08:55.430289 25236 ts_tablet_manager.cc:1367] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 14:08:55.434298 25235 raft_consensus.cc:348] T 304de776104a4ab382e9274e3006be59 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:55.434480 25237 tablet_bootstrap.cc:492] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:08:55.434648 25235 raft_consensus.cc:374] T 304de776104a4ab382e9274e3006be59 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.434662 25237 ts_tablet_manager.cc:1367] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 14:08:55.434744 25235 raft_consensus.cc:729] T 304de776104a4ab382e9274e3006be59 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.435300 25235 consensus_queue.cc:260] T 304de776104a4ab382e9274e3006be59 P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:55.436259 25235 ts_tablet_manager.cc:1398] T 304de776104a4ab382e9274e3006be59 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20240501 14:08:55.436651 25235 tablet_bootstrap.cc:492] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:08:55.437664 25235 tablet_bootstrap.cc:654] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.438843 25235 tablet_bootstrap.cc:492] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:08:55.438926 25235 ts_tablet_manager.cc:1367] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:55.439415 25235 raft_consensus.cc:348] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:55.439510 25235 raft_consensus.cc:374] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.439538 25235 raft_consensus.cc:729] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.439497 25237 raft_consensus.cc:348] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:55.439695 25237 raft_consensus.cc:374] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.439671 25235 consensus_queue.cc:260] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:55.439739 25237 raft_consensus.cc:729] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.440009 25235 ts_tablet_manager.cc:1398] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:55.440173 25237 consensus_queue.cc:260] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:55.441082 25237 ts_tablet_manager.cc:1398] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 14:08:55.441665 25236 raft_consensus.cc:348] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:55.441907 25236 raft_consensus.cc:374] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.441979 25236 raft_consensus.cc:729] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.442405 25236 consensus_queue.cc:260] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:55.443289 25236 ts_tablet_manager.cc:1398] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.013s user 0.003s sys 0.002s
I20240501 14:08:55.446697 25244 tablet_bootstrap.cc:492] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:08:55.446692 25241 tablet_bootstrap.cc:492] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:08:55.448741 25248 tablet_bootstrap.cc:492] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:08:55.448902 25244 tablet_bootstrap.cc:654] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.448902 25241 tablet_bootstrap.cc:654] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.449719 25244 log.cc:824] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:55.449718 25241 log.cc:824] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:55.450634 25248 tablet_bootstrap.cc:654] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.451229 25248 log.cc:824] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:55.451431 25241 tablet_bootstrap.cc:492] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:08:55.451431 25244 tablet_bootstrap.cc:492] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:08:55.451607 25244 ts_tablet_manager.cc:1367] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 14:08:55.451607 25241 ts_tablet_manager.cc:1367] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 14:08:55.452550 25248 tablet_bootstrap.cc:492] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:08:55.452701 25248 ts_tablet_manager.cc:1367] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:08:55.455441 25244 raft_consensus.cc:348] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:55.455441 25241 raft_consensus.cc:348] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:55.455646 25241 raft_consensus.cc:374] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.455714 25241 raft_consensus.cc:729] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.455729 25244 raft_consensus.cc:374] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.455823 25244 raft_consensus.cc:729] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.455750 25248 raft_consensus.cc:348] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:55.456018 25248 raft_consensus.cc:374] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.456075 25248 raft_consensus.cc:729] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.456197 25241 consensus_queue.cc:260] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:55.456197 25244 consensus_queue.cc:260] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:55.456455 25248 consensus_queue.cc:260] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:55.457103 25244 ts_tablet_manager.cc:1398] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20240501 14:08:55.457106 25241 ts_tablet_manager.cc:1398] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20240501 14:08:55.457346 25241 tablet_bootstrap.cc:492] T a5fd0fa373d74cd184b6626c559e3ee0 P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:08:55.457369 25244 tablet_bootstrap.cc:492] T 304de776104a4ab382e9274e3006be59 P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:08:55.458299 25244 tablet_bootstrap.cc:654] T 304de776104a4ab382e9274e3006be59 P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.458299 25241 tablet_bootstrap.cc:654] T a5fd0fa373d74cd184b6626c559e3ee0 P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.458317 25248 ts_tablet_manager.cc:1398] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:08:55.459544 25241 tablet_bootstrap.cc:492] T a5fd0fa373d74cd184b6626c559e3ee0 P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:08:55.459632 25241 ts_tablet_manager.cc:1367] T a5fd0fa373d74cd184b6626c559e3ee0 P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:55.460155 25241 raft_consensus.cc:348] T a5fd0fa373d74cd184b6626c559e3ee0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:55.460250 25241 raft_consensus.cc:374] T a5fd0fa373d74cd184b6626c559e3ee0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.460283 25241 raft_consensus.cc:729] T a5fd0fa373d74cd184b6626c559e3ee0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.460408 25241 consensus_queue.cc:260] T a5fd0fa373d74cd184b6626c559e3ee0 P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:55.460570 25244 tablet_bootstrap.cc:492] T 304de776104a4ab382e9274e3006be59 P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:08:55.460649 25244 ts_tablet_manager.cc:1367] T 304de776104a4ab382e9274e3006be59 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:55.460644 25241 ts_tablet_manager.cc:1398] T a5fd0fa373d74cd184b6626c559e3ee0 P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:55.461164 25244 raft_consensus.cc:348] T 304de776104a4ab382e9274e3006be59 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:55.461274 25244 raft_consensus.cc:374] T 304de776104a4ab382e9274e3006be59 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.461308 25244 raft_consensus.cc:729] T 304de776104a4ab382e9274e3006be59 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.461413 25244 consensus_queue.cc:260] T 304de776104a4ab382e9274e3006be59 P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:55.461650 25244 ts_tablet_manager.cc:1398] T 304de776104a4ab382e9274e3006be59 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 14:08:55.468178 24945 tablet.cc:2369] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:55.498504 25254 raft_consensus.cc:482] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:55.498764 25254 raft_consensus.cc:504] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:55.500346 25254 leader_election.cc:288] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345), 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635)
I20240501 14:08:55.505090 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5fd0fa373d74cd184b6626c559e3ee0" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79" is_pre_election: true
I20240501 14:08:55.505373 25030 raft_consensus.cc:2454] T a5fd0fa373d74cd184b6626c559e3ee0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 0.
I20240501 14:08:55.505791 24831 leader_election.cc:302] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e375044d63649c097759c4b667eff79, c2691129fd82431bb867cb1307e009b6; no voters:
I20240501 14:08:55.506039 25254 raft_consensus.cc:2781] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:55.506006 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5fd0fa373d74cd184b6626c559e3ee0" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3" is_pre_election: true
I20240501 14:08:55.506122 25254 raft_consensus.cc:482] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:55.506170 25254 raft_consensus.cc:3037] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.506294 24634 raft_consensus.cc:2454] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 0.
I20240501 14:08:55.507104 25254 raft_consensus.cc:504] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:55.507413 25254 leader_election.cc:288] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 election: Requested vote from peers f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345), 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635)
I20240501 14:08:55.507761 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5fd0fa373d74cd184b6626c559e3ee0" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3"
I20240501 14:08:55.507813 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5fd0fa373d74cd184b6626c559e3ee0" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79"
I20240501 14:08:55.507874 24634 raft_consensus.cc:3037] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.507942 25030 raft_consensus.cc:3037] T a5fd0fa373d74cd184b6626c559e3ee0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.508719 24634 raft_consensus.cc:2454] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 1.
I20240501 14:08:55.508846 25030 raft_consensus.cc:2454] T a5fd0fa373d74cd184b6626c559e3ee0 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 1.
I20240501 14:08:55.509128 24831 leader_election.cc:302] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2691129fd82431bb867cb1307e009b6, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:55.509322 25254 raft_consensus.cc:2781] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:55.509521 25254 raft_consensus.cc:686] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [term 1 LEADER]: Becoming Leader. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Running, Role: LEADER
I20240501 14:08:55.509847 25254 consensus_queue.cc:237] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:55.512799 24358 catalog_manager.cc:5484] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2691129fd82431bb867cb1307e009b6 (127.23.191.68). New cstate: current_term: 1 leader_uuid: "c2691129fd82431bb867cb1307e009b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:55.514851 25247 raft_consensus.cc:482] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:55.515017 25247 raft_consensus.cc:504] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:55.515996 25247 leader_election.cc:288] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:08:55.520529 24766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304de776104a4ab382e9274e3006be59" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" is_pre_election: true
I20240501 14:08:55.520681 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304de776104a4ab382e9274e3006be59" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:08:55.520802 24766 raft_consensus.cc:2454] T 304de776104a4ab382e9274e3006be59 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:08:55.520977 24502 raft_consensus.cc:2454] T 304de776104a4ab382e9274e3006be59 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:08:55.521276 25095 leader_election.cc:302] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 786f81b7c57f452e9b463f721fc3e865; no voters:
I20240501 14:08:55.521515 25247 raft_consensus.cc:2781] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:55.521595 25247 raft_consensus.cc:482] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:55.521656 25247 raft_consensus.cc:3037] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.522472 25247 raft_consensus.cc:504] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:55.522763 25247 leader_election.cc:288] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Requested vote from peers 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:08:55.523056 24766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304de776104a4ab382e9274e3006be59" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
I20240501 14:08:55.523113 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304de776104a4ab382e9274e3006be59" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:08:55.523195 24766 raft_consensus.cc:3037] T 304de776104a4ab382e9274e3006be59 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.523226 24502 raft_consensus.cc:3037] T 304de776104a4ab382e9274e3006be59 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.523996 24766 raft_consensus.cc:2454] T 304de776104a4ab382e9274e3006be59 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:08:55.524070 24502 raft_consensus.cc:2454] T 304de776104a4ab382e9274e3006be59 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:08:55.524278 25095 leader_election.cc:302] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 786f81b7c57f452e9b463f721fc3e865; no voters:
I20240501 14:08:55.524428 25247 raft_consensus.cc:2781] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:55.524641 25247 raft_consensus.cc:686] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [term 1 LEADER]: Becoming Leader. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Running, Role: LEADER
I20240501 14:08:55.524945 25247 consensus_queue.cc:237] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:55.527668 24358 catalog_manager.cc:5484] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70). New cstate: current_term: 1 leader_uuid: "786f81b7c57f452e9b463f721fc3e865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: HEALTHY } } }
W20240501 14:08:55.532279 24813 tablet.cc:2369] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:08:55.583151 25210 tablet.cc:2369] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:08:55.599342 24681 tablet.cc:2369] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:08:55.642854 25077 tablet.cc:2369] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:55.676085 25242 raft_consensus.cc:482] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:55.676229 25242 raft_consensus.cc:504] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:55.677361 25242 leader_election.cc:288] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635)
W20240501 14:08:55.678048 24549 tablet.cc:2369] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:55.681576 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29231eab3ff84873a05a65ad6513c4e5" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:08:55.681721 24502 raft_consensus.cc:2454] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 0.
I20240501 14:08:55.682092 24700 leader_election.cc:302] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:08:55.682317 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29231eab3ff84873a05a65ad6513c4e5" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79" is_pre_election: true
I20240501 14:08:55.682348 25242 raft_consensus.cc:2781] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:55.682420 25242 raft_consensus.cc:482] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:55.682462 25242 raft_consensus.cc:3037] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.682446 25030 raft_consensus.cc:2454] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 0.
I20240501 14:08:55.683285 25242 raft_consensus.cc:504] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:55.683665 25242 leader_election.cc:288] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 election: Requested vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635)
I20240501 14:08:55.683986 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29231eab3ff84873a05a65ad6513c4e5" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:08:55.683993 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29231eab3ff84873a05a65ad6513c4e5" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79"
I20240501 14:08:55.684087 25030 raft_consensus.cc:3037] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.684092 24502 raft_consensus.cc:3037] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.684878 24502 raft_consensus.cc:2454] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 1.
I20240501 14:08:55.684907 25030 raft_consensus.cc:2454] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 1.
I20240501 14:08:55.685134 24700 leader_election.cc:302] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:08:55.685292 25242 raft_consensus.cc:2781] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:55.685384 25242 raft_consensus.cc:686] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 LEADER]: Becoming Leader. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Running, Role: LEADER
I20240501 14:08:55.685683 25242 consensus_queue.cc:237] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:55.688709 24358 catalog_manager.cc:5484] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67). New cstate: current_term: 1 leader_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: UNKNOWN } } }
W20240501 14:08:55.807018 25268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:55.807358 25268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:55.810487 25268 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 14:08:55.911265 24766 raft_consensus.cc:1261] T 304de776104a4ab382e9274e3006be59 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:55.911401 24502 raft_consensus.cc:1261] T 304de776104a4ab382e9274e3006be59 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:55.912065 25263 consensus_queue.cc:1035] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:55.912786 25247 consensus_queue.cc:1035] T 304de776104a4ab382e9274e3006be59 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:55.914741 25030 raft_consensus.cc:1261] T a5fd0fa373d74cd184b6626c559e3ee0 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Refusing update from remote peer c2691129fd82431bb867cb1307e009b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:55.916014 25254 consensus_queue.cc:1035] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:55.917281 24634 raft_consensus.cc:1261] T a5fd0fa373d74cd184b6626c559e3ee0 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Refusing update from remote peer c2691129fd82431bb867cb1307e009b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:55.917837 25254 consensus_queue.cc:1035] T a5fd0fa373d74cd184b6626c559e3ee0 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:55.919381 24501 raft_consensus.cc:1261] T 29231eab3ff84873a05a65ad6513c4e5 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Refusing update from remote peer 319fbe37b2ce4434a8d2fe879ca920a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:55.919379 25029 raft_consensus.cc:1261] T 29231eab3ff84873a05a65ad6513c4e5 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Refusing update from remote peer 319fbe37b2ce4434a8d2fe879ca920a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:55.919845 25245 consensus_queue.cc:1035] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:55.920219 25245 consensus_queue.cc:1035] T 29231eab3ff84873a05a65ad6513c4e5 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:55.930526 25308 mvcc.cc:205] Tried to move back new op lower bound from 7022889107121614848 to 7022889106171043840. Current Snapshot: MvccSnapshot[applied={T|T < 7022889107121614848}]
I20240501 14:08:55.930976 25300 mvcc.cc:205] Tried to move back new op lower bound from 7022889107121614848 to 7022889106171043840. Current Snapshot: MvccSnapshot[applied={T|T < 7022889107121614848}]
I20240501 14:08:55.934439 25299 mvcc.cc:205] Tried to move back new op lower bound from 7022889107098697728 to 7022889105450713088. Current Snapshot: MvccSnapshot[applied={T|T < 7022889107098697728}]
I20240501 14:08:55.938043 25311 mvcc.cc:205] Tried to move back new op lower bound from 7022889107098697728 to 7022889105450713088. Current Snapshot: MvccSnapshot[applied={T|T < 7022889107098697728}]
I20240501 14:08:55.939579 25298 mvcc.cc:205] Tried to move back new op lower bound from 7022889107085668352 to 7022889105512378368. Current Snapshot: MvccSnapshot[applied={T|T < 7022889107085668352}]
I20240501 14:08:55.949573 25306 mvcc.cc:205] Tried to move back new op lower bound from 7022889107098697728 to 7022889105450713088. Current Snapshot: MvccSnapshot[applied={T|T < 7022889107098697728}]
INSERT report
rows total: 2000
time total: 130.991 ms
time per row: 0.0654956 ms
I20240501 14:08:55.990144 24357 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43068:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:55.995517 25143 tablet_service.cc:1467] Processing CreateTablet for tablet ffb185b33baf4a0985355820d312f474 (DEFAULT_TABLE table=rebalance_test_table_1 [id=95289175e6b1437cbe0541e2f3cd72be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.995517 24878 tablet_service.cc:1467] Processing CreateTablet for tablet ffb185b33baf4a0985355820d312f474 (DEFAULT_TABLE table=rebalance_test_table_1 [id=95289175e6b1437cbe0541e2f3cd72be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.995828 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb185b33baf4a0985355820d312f474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.995828 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb185b33baf4a0985355820d312f474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.996336 25010 tablet_service.cc:1467] Processing CreateTablet for tablet 451c6589d9444bbfb539d82d65fa3be6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=95289175e6b1437cbe0541e2f3cd72be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.996631 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451c6589d9444bbfb539d82d65fa3be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.997476 24482 tablet_service.cc:1467] Processing CreateTablet for tablet 451c6589d9444bbfb539d82d65fa3be6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=95289175e6b1437cbe0541e2f3cd72be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.997761 24482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451c6589d9444bbfb539d82d65fa3be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.998253 24614 tablet_service.cc:1467] Processing CreateTablet for tablet ffb185b33baf4a0985355820d312f474 (DEFAULT_TABLE table=rebalance_test_table_1 [id=95289175e6b1437cbe0541e2f3cd72be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.998548 24614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb185b33baf4a0985355820d312f474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.999233 25333 tablet_bootstrap.cc:492] T 451c6589d9444bbfb539d82d65fa3be6 P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:08:55.999984 25334 tablet_bootstrap.cc:492] T 451c6589d9444bbfb539d82d65fa3be6 P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:08:56.000125 25333 tablet_bootstrap.cc:654] T 451c6589d9444bbfb539d82d65fa3be6 P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.000641 24613 tablet_service.cc:1467] Processing CreateTablet for tablet 3973da211ea444b18da475e7e70c7e68 (DEFAULT_TABLE table=rebalance_test_table_1 [id=95289175e6b1437cbe0541e2f3cd72be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.000908 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3973da211ea444b18da475e7e70c7e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.000926 25334 tablet_bootstrap.cc:654] T 451c6589d9444bbfb539d82d65fa3be6 P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.000950 25335 tablet_bootstrap.cc:492] T ffb185b33baf4a0985355820d312f474 P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:08:56.001906 25334 tablet_bootstrap.cc:492] T 451c6589d9444bbfb539d82d65fa3be6 P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:08:56.001936 25335 tablet_bootstrap.cc:654] T ffb185b33baf4a0985355820d312f474 P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.001986 25334 ts_tablet_manager.cc:1367] T 451c6589d9444bbfb539d82d65fa3be6 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:08:56.002416 25334 raft_consensus.cc:348] T 451c6589d9444bbfb539d82d65fa3be6 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:56.002521 25334 raft_consensus.cc:374] T 451c6589d9444bbfb539d82d65fa3be6 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.002552 25334 raft_consensus.cc:729] T 451c6589d9444bbfb539d82d65fa3be6 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.002672 25334 consensus_queue.cc:260] T 451c6589d9444bbfb539d82d65fa3be6 P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:56.002940 25334 ts_tablet_manager.cc:1398] T 451c6589d9444bbfb539d82d65fa3be6 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:56.002949 25335 tablet_bootstrap.cc:492] T ffb185b33baf4a0985355820d312f474 P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:08:56.003023 25335 ts_tablet_manager.cc:1367] T ffb185b33baf4a0985355820d312f474 P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:08:56.003069 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 451c6589d9444bbfb539d82d65fa3be6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=95289175e6b1437cbe0541e2f3cd72be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.003335 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451c6589d9444bbfb539d82d65fa3be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.003572 25335 raft_consensus.cc:348] T ffb185b33baf4a0985355820d312f474 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:56.003662 25335 raft_consensus.cc:374] T ffb185b33baf4a0985355820d312f474 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.003692 25335 raft_consensus.cc:729] T ffb185b33baf4a0985355820d312f474 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.003809 25335 consensus_queue.cc:260] T ffb185b33baf4a0985355820d312f474 P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:56.004034 25335 ts_tablet_manager.cc:1398] T ffb185b33baf4a0985355820d312f474 P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:56.004159 25335 tablet_bootstrap.cc:492] T 3973da211ea444b18da475e7e70c7e68 P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:08:56.005455 25335 tablet_bootstrap.cc:654] T 3973da211ea444b18da475e7e70c7e68 P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.005606 25143 tablet_service.cc:1467] Processing CreateTablet for tablet 3973da211ea444b18da475e7e70c7e68 (DEFAULT_TABLE table=rebalance_test_table_1 [id=95289175e6b1437cbe0541e2f3cd72be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.005896 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3973da211ea444b18da475e7e70c7e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.006675 25335 tablet_bootstrap.cc:492] T 3973da211ea444b18da475e7e70c7e68 P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:08:56.006676 25333 tablet_bootstrap.cc:492] T 451c6589d9444bbfb539d82d65fa3be6 P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:08:56.006758 25335 ts_tablet_manager.cc:1367] T 3973da211ea444b18da475e7e70c7e68 P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:08:56.006760 25333 ts_tablet_manager.cc:1367] T 451c6589d9444bbfb539d82d65fa3be6 P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:08:56.007161 25335 raft_consensus.cc:348] T 3973da211ea444b18da475e7e70c7e68 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.007161 25333 raft_consensus.cc:348] T 451c6589d9444bbfb539d82d65fa3be6 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:56.007258 25335 raft_consensus.cc:374] T 3973da211ea444b18da475e7e70c7e68 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.007258 25333 raft_consensus.cc:374] T 451c6589d9444bbfb539d82d65fa3be6 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.007288 25333 raft_consensus.cc:729] T 451c6589d9444bbfb539d82d65fa3be6 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.007288 25335 raft_consensus.cc:729] T 3973da211ea444b18da475e7e70c7e68 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.007427 25335 consensus_queue.cc:260] T 3973da211ea444b18da475e7e70c7e68 P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.007434 25333 consensus_queue.cc:260] T 451c6589d9444bbfb539d82d65fa3be6 P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:56.007671 25335 ts_tablet_manager.cc:1398] T 3973da211ea444b18da475e7e70c7e68 P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:56.007830 24878 tablet_service.cc:1467] Processing CreateTablet for tablet 3973da211ea444b18da475e7e70c7e68 (DEFAULT_TABLE table=rebalance_test_table_1 [id=95289175e6b1437cbe0541e2f3cd72be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.008100 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3973da211ea444b18da475e7e70c7e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.008770 25331 tablet_bootstrap.cc:492] T ffb185b33baf4a0985355820d312f474 P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:08:56.008894 25333 ts_tablet_manager.cc:1398] T 451c6589d9444bbfb539d82d65fa3be6 P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:56.009673 25331 tablet_bootstrap.cc:654] T ffb185b33baf4a0985355820d312f474 P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.010361 25340 tablet_bootstrap.cc:492] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:08:56.010733 25331 tablet_bootstrap.cc:492] T ffb185b33baf4a0985355820d312f474 P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:08:56.010809 25331 ts_tablet_manager.cc:1367] T ffb185b33baf4a0985355820d312f474 P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:56.011251 25340 tablet_bootstrap.cc:654] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.011304 25331 raft_consensus.cc:348] T ffb185b33baf4a0985355820d312f474 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:56.011413 25331 raft_consensus.cc:374] T ffb185b33baf4a0985355820d312f474 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.011467 25331 raft_consensus.cc:729] T ffb185b33baf4a0985355820d312f474 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.011616 25331 consensus_queue.cc:260] T ffb185b33baf4a0985355820d312f474 P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:56.011976 25332 tablet_bootstrap.cc:492] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:08:56.012228 25331 ts_tablet_manager.cc:1398] T ffb185b33baf4a0985355820d312f474 P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:56.012411 25331 tablet_bootstrap.cc:492] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:08:56.012444 25340 tablet_bootstrap.cc:492] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:08:56.012559 25340 ts_tablet_manager.cc:1367] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:56.012902 25332 tablet_bootstrap.cc:654] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.013046 25340 raft_consensus.cc:348] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:56.013223 25340 raft_consensus.cc:374] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.013286 25331 tablet_bootstrap.cc:654] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.013305 25340 raft_consensus.cc:729] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.013496 25340 consensus_queue.cc:260] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:56.013828 25340 ts_tablet_manager.cc:1398] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:56.014528 25332 tablet_bootstrap.cc:492] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:08:56.014612 25332 ts_tablet_manager.cc:1367] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:56.015035 25331 tablet_bootstrap.cc:492] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:08:56.015108 25331 ts_tablet_manager.cc:1367] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:08:56.015146 25332 raft_consensus.cc:348] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:56.015254 25332 raft_consensus.cc:374] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.015285 25332 raft_consensus.cc:729] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.015405 25332 consensus_queue.cc:260] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:56.015437 25331 raft_consensus.cc:348] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.015527 25331 raft_consensus.cc:374] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.015636 25332 ts_tablet_manager.cc:1398] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:56.015761 25332 tablet_bootstrap.cc:492] T 3973da211ea444b18da475e7e70c7e68 P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:08:56.015823 25245 raft_consensus.cc:482] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:56.015892 25245 raft_consensus.cc:504] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:56.016230 25245 leader_election.cc:288] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:08:56.016667 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451c6589d9444bbfb539d82d65fa3be6" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79" is_pre_election: true
I20240501 14:08:56.016815 25332 tablet_bootstrap.cc:654] T 3973da211ea444b18da475e7e70c7e68 P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.016762 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451c6589d9444bbfb539d82d65fa3be6" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:08:56.016799 25029 raft_consensus.cc:2454] T 451c6589d9444bbfb539d82d65fa3be6 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 0.
I20240501 14:08:56.016935 24502 raft_consensus.cc:2454] T 451c6589d9444bbfb539d82d65fa3be6 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 0.
I20240501 14:08:56.017122 24699 leader_election.cc:302] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:08:56.017280 25245 raft_consensus.cc:2781] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:56.017344 25245 raft_consensus.cc:482] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:56.017382 25245 raft_consensus.cc:3037] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.015558 25331 raft_consensus.cc:729] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.017935 25331 consensus_queue.cc:260] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.018026 25332 tablet_bootstrap.cc:492] T 3973da211ea444b18da475e7e70c7e68 P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:08:56.018100 25332 ts_tablet_manager.cc:1367] T 3973da211ea444b18da475e7e70c7e68 P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:56.018325 25331 ts_tablet_manager.cc:1398] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:08:56.018362 25245 raft_consensus.cc:504] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:56.018501 25332 raft_consensus.cc:348] T 3973da211ea444b18da475e7e70c7e68 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.018594 25332 raft_consensus.cc:374] T 3973da211ea444b18da475e7e70c7e68 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.018620 25332 raft_consensus.cc:729] T 3973da211ea444b18da475e7e70c7e68 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.018699 25245 leader_election.cc:288] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 election: Requested vote from peers 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:08:56.018733 25332 consensus_queue.cc:260] T 3973da211ea444b18da475e7e70c7e68 P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.018926 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451c6589d9444bbfb539d82d65fa3be6" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79"
I20240501 14:08:56.018968 25332 ts_tablet_manager.cc:1398] T 3973da211ea444b18da475e7e70c7e68 P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:56.019035 25029 raft_consensus.cc:3037] T 451c6589d9444bbfb539d82d65fa3be6 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.019130 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451c6589d9444bbfb539d82d65fa3be6" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:08:56.019239 24502 raft_consensus.cc:3037] T 451c6589d9444bbfb539d82d65fa3be6 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.019985 25029 raft_consensus.cc:2454] T 451c6589d9444bbfb539d82d65fa3be6 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 1.
I20240501 14:08:56.020120 24502 raft_consensus.cc:2454] T 451c6589d9444bbfb539d82d65fa3be6 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 1.
I20240501 14:08:56.020377 24699 leader_election.cc:302] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:08:56.020680 25245 raft_consensus.cc:2781] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:56.020787 25245 raft_consensus.cc:686] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 LEADER]: Becoming Leader. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Running, Role: LEADER
I20240501 14:08:56.020956 25245 consensus_queue.cc:237] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:56.022428 24360 catalog_manager.cc:5484] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67). New cstate: current_term: 1 leader_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:56.188110 25263 raft_consensus.cc:482] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:56.188246 25263 raft_consensus.cc:504] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:56.188717 25263 leader_election.cc:288] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:08:56.192148 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb185b33baf4a0985355820d312f474" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6" is_pre_election: true
I20240501 14:08:56.192163 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb185b33baf4a0985355820d312f474" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3" is_pre_election: true
I20240501 14:08:56.192305 24634 raft_consensus.cc:2454] T ffb185b33baf4a0985355820d312f474 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:08:56.192380 24898 raft_consensus.cc:2454] T ffb185b33baf4a0985355820d312f474 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:08:56.192577 25095 leader_election.cc:302] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:56.192742 25263 raft_consensus.cc:2781] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:56.192802 25263 raft_consensus.cc:482] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:56.192852 25263 raft_consensus.cc:3037] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.193621 25263 raft_consensus.cc:504] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:56.193933 25263 leader_election.cc:288] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Requested vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:08:56.194269 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb185b33baf4a0985355820d312f474" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3"
I20240501 14:08:56.194263 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb185b33baf4a0985355820d312f474" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6"
I20240501 14:08:56.194391 24634 raft_consensus.cc:3037] T ffb185b33baf4a0985355820d312f474 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.194391 24898 raft_consensus.cc:3037] T ffb185b33baf4a0985355820d312f474 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.195366 24634 raft_consensus.cc:2454] T ffb185b33baf4a0985355820d312f474 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:08:56.195647 25095 leader_election.cc:302] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:56.195685 24898 raft_consensus.cc:2454] T ffb185b33baf4a0985355820d312f474 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:08:56.195819 25263 raft_consensus.cc:2781] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:56.195917 25263 raft_consensus.cc:686] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [term 1 LEADER]: Becoming Leader. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Running, Role: LEADER
I20240501 14:08:56.196063 25263 consensus_queue.cc:237] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:56.197669 24360 catalog_manager.cc:5484] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70). New cstate: current_term: 1 leader_uuid: "786f81b7c57f452e9b463f721fc3e865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:56.223631 25259 raft_consensus.cc:482] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:56.223776 25259 raft_consensus.cc:504] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.224277 25259 leader_election.cc:288] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345), 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343)
I20240501 14:08:56.224778 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3973da211ea444b18da475e7e70c7e68" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3" is_pre_election: true
I20240501 14:08:56.224962 24634 raft_consensus.cc:2454] T 3973da211ea444b18da475e7e70c7e68 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 0.
I20240501 14:08:56.225255 24831 leader_election.cc:302] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2691129fd82431bb867cb1307e009b6, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:56.225455 25259 raft_consensus.cc:2781] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:56.225523 25259 raft_consensus.cc:482] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:56.225561 25259 raft_consensus.cc:3037] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.226545 25259 raft_consensus.cc:504] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.226951 25259 leader_election.cc:288] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 election: Requested vote from peers f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345), 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343)
I20240501 14:08:56.227304 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3973da211ea444b18da475e7e70c7e68" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3"
I20240501 14:08:56.227437 24634 raft_consensus.cc:3037] T 3973da211ea444b18da475e7e70c7e68 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.228425 24634 raft_consensus.cc:2454] T 3973da211ea444b18da475e7e70c7e68 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 1.
I20240501 14:08:56.228711 24831 leader_election.cc:302] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2691129fd82431bb867cb1307e009b6, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:56.228914 25259 raft_consensus.cc:2781] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:56.228804 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3973da211ea444b18da475e7e70c7e68" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865"
I20240501 14:08:56.228807 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3973da211ea444b18da475e7e70c7e68" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865" is_pre_election: true
I20240501 14:08:56.228996 25259 raft_consensus.cc:686] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [term 1 LEADER]: Becoming Leader. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Running, Role: LEADER
I20240501 14:08:56.229090 25163 raft_consensus.cc:2454] T 3973da211ea444b18da475e7e70c7e68 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 0.
I20240501 14:08:56.229136 25259 consensus_queue.cc:237] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.231272 24358 catalog_manager.cc:5484] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2691129fd82431bb867cb1307e009b6 (127.23.191.68). New cstate: current_term: 1 leader_uuid: "c2691129fd82431bb867cb1307e009b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: HEALTHY } } }
W20240501 14:08:56.394338 25353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:56.394651 25353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:56.397804 25353 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:08:56.470700 25245 consensus_queue.cc:1035] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:56.484473 25242 consensus_queue.cc:1035] T 451c6589d9444bbfb539d82d65fa3be6 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:56.488983 24634 raft_consensus.cc:1261] T ffb185b33baf4a0985355820d312f474 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:56.489482 25247 consensus_queue.cc:1035] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:56.489699 25163 raft_consensus.cc:3037] T 3973da211ea444b18da475e7e70c7e68 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.489768 24898 raft_consensus.cc:1261] T ffb185b33baf4a0985355820d312f474 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:56.490272 25247 consensus_queue.cc:1035] T ffb185b33baf4a0985355820d312f474 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:56.490797 25163 raft_consensus.cc:1261] T 3973da211ea444b18da475e7e70c7e68 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Refusing update from remote peer c2691129fd82431bb867cb1307e009b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:56.491570 24633 raft_consensus.cc:1261] T 3973da211ea444b18da475e7e70c7e68 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Refusing update from remote peer c2691129fd82431bb867cb1307e009b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:56.491992 25259 consensus_queue.cc:1035] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:56.494345 25254 consensus_queue.cc:1035] T 3973da211ea444b18da475e7e70c7e68 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 109.031 ms
time per row: 0.0545154 ms
I20240501 14:08:56.552646 24357 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43068:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:56.558493 25010 tablet_service.cc:1467] Processing CreateTablet for tablet 430ccd0aed3948f9a91560ace381d83c (DEFAULT_TABLE table=rebalance_test_table_2 [id=15b47c1cc597474a8ea13856b75b9224]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.558555 24482 tablet_service.cc:1467] Processing CreateTablet for tablet 430ccd0aed3948f9a91560ace381d83c (DEFAULT_TABLE table=rebalance_test_table_2 [id=15b47c1cc597474a8ea13856b75b9224]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.558830 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430ccd0aed3948f9a91560ace381d83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.558861 24482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430ccd0aed3948f9a91560ace381d83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.558840 24878 tablet_service.cc:1467] Processing CreateTablet for tablet 8060ab0e7d594d6d8dd1d789844062a7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=15b47c1cc597474a8ea13856b75b9224]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.558954 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 430ccd0aed3948f9a91560ace381d83c (DEFAULT_TABLE table=rebalance_test_table_2 [id=15b47c1cc597474a8ea13856b75b9224]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.559082 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8060ab0e7d594d6d8dd1d789844062a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.559194 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430ccd0aed3948f9a91560ace381d83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.561445 25419 tablet_bootstrap.cc:492] T 8060ab0e7d594d6d8dd1d789844062a7 P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:08:56.561439 25143 tablet_service.cc:1467] Processing CreateTablet for tablet 8060ab0e7d594d6d8dd1d789844062a7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=15b47c1cc597474a8ea13856b75b9224]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.561477 24613 tablet_service.cc:1467] Processing CreateTablet for tablet 8060ab0e7d594d6d8dd1d789844062a7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=15b47c1cc597474a8ea13856b75b9224]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.561693 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8060ab0e7d594d6d8dd1d789844062a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.561743 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8060ab0e7d594d6d8dd1d789844062a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.562264 24746 tablet_service.cc:1467] Processing CreateTablet for tablet c4ae6674f0534b5eafc91d6579849ff0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=15b47c1cc597474a8ea13856b75b9224]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.562398 25419 tablet_bootstrap.cc:654] T 8060ab0e7d594d6d8dd1d789844062a7 P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.562477 24746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ae6674f0534b5eafc91d6579849ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.563071 25420 tablet_bootstrap.cc:492] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:08:56.563918 25420 tablet_bootstrap.cc:654] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.564193 25423 tablet_bootstrap.cc:492] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:08:56.564397 24482 tablet_service.cc:1467] Processing CreateTablet for tablet c4ae6674f0534b5eafc91d6579849ff0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=15b47c1cc597474a8ea13856b75b9224]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.564622 24482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ae6674f0534b5eafc91d6579849ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.564744 25010 tablet_service.cc:1467] Processing CreateTablet for tablet c4ae6674f0534b5eafc91d6579849ff0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=15b47c1cc597474a8ea13856b75b9224]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.565037 25421 tablet_bootstrap.cc:492] T 430ccd0aed3948f9a91560ace381d83c P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:08:56.565048 25423 tablet_bootstrap.cc:654] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.565387 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ae6674f0534b5eafc91d6579849ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.565678 25424 tablet_bootstrap.cc:492] T 8060ab0e7d594d6d8dd1d789844062a7 P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:08:56.565857 25421 tablet_bootstrap.cc:654] T 430ccd0aed3948f9a91560ace381d83c P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.566663 25424 tablet_bootstrap.cc:654] T 8060ab0e7d594d6d8dd1d789844062a7 P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.566895 25420 tablet_bootstrap.cc:492] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:08:56.566975 25420 ts_tablet_manager.cc:1367] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:08:56.567329 25421 tablet_bootstrap.cc:492] T 430ccd0aed3948f9a91560ace381d83c P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:08:56.567446 25421 ts_tablet_manager.cc:1367] T 430ccd0aed3948f9a91560ace381d83c P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:56.567406 25420 raft_consensus.cc:348] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.567502 25420 raft_consensus.cc:374] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.567536 25420 raft_consensus.cc:729] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.567688 25423 tablet_bootstrap.cc:492] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:08:56.567653 25420 consensus_queue.cc:260] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.567916 25420 ts_tablet_manager.cc:1398] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:56.567880 25421 raft_consensus.cc:348] T 430ccd0aed3948f9a91560ace381d83c P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.567970 25421 raft_consensus.cc:374] T 430ccd0aed3948f9a91560ace381d83c P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.567997 25421 raft_consensus.cc:729] T 430ccd0aed3948f9a91560ace381d83c P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.568070 25420 tablet_bootstrap.cc:492] T c4ae6674f0534b5eafc91d6579849ff0 P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:08:56.568107 25421 consensus_queue.cc:260] T 430ccd0aed3948f9a91560ace381d83c P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.568361 25421 ts_tablet_manager.cc:1398] T 430ccd0aed3948f9a91560ace381d83c P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:56.568527 25421 tablet_bootstrap.cc:492] T c4ae6674f0534b5eafc91d6579849ff0 P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:08:56.568528 25425 tablet_bootstrap.cc:492] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:08:56.568890 25423 ts_tablet_manager.cc:1367] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:08:56.569133 25420 tablet_bootstrap.cc:654] T c4ae6674f0534b5eafc91d6579849ff0 P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.569324 25421 tablet_bootstrap.cc:654] T c4ae6674f0534b5eafc91d6579849ff0 P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.569336 25423 raft_consensus.cc:348] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.569432 25423 raft_consensus.cc:374] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.569459 25423 raft_consensus.cc:729] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.569603 25423 consensus_queue.cc:260] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.570343 25420 tablet_bootstrap.cc:492] T c4ae6674f0534b5eafc91d6579849ff0 P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:08:56.570420 25420 ts_tablet_manager.cc:1367] T c4ae6674f0534b5eafc91d6579849ff0 P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:56.570447 25421 tablet_bootstrap.cc:492] T c4ae6674f0534b5eafc91d6579849ff0 P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:08:56.570456 25423 ts_tablet_manager.cc:1398] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:08:56.570510 25421 ts_tablet_manager.cc:1367] T c4ae6674f0534b5eafc91d6579849ff0 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:08:56.570816 25420 raft_consensus.cc:348] T c4ae6674f0534b5eafc91d6579849ff0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.570858 25421 raft_consensus.cc:348] T c4ae6674f0534b5eafc91d6579849ff0 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.570921 25420 raft_consensus.cc:374] T c4ae6674f0534b5eafc91d6579849ff0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.570931 25421 raft_consensus.cc:374] T c4ae6674f0534b5eafc91d6579849ff0 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.570951 25420 raft_consensus.cc:729] T c4ae6674f0534b5eafc91d6579849ff0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.570960 25421 raft_consensus.cc:729] T c4ae6674f0534b5eafc91d6579849ff0 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.571060 25420 consensus_queue.cc:260] T c4ae6674f0534b5eafc91d6579849ff0 P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.571060 25421 consensus_queue.cc:260] T c4ae6674f0534b5eafc91d6579849ff0 P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.571312 25421 ts_tablet_manager.cc:1398] T c4ae6674f0534b5eafc91d6579849ff0 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:56.571321 25420 ts_tablet_manager.cc:1398] T c4ae6674f0534b5eafc91d6579849ff0 P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:56.571801 25424 tablet_bootstrap.cc:492] T 8060ab0e7d594d6d8dd1d789844062a7 P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:08:56.571964 25424 ts_tablet_manager.cc:1367] T 8060ab0e7d594d6d8dd1d789844062a7 P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:08:56.573261 25424 raft_consensus.cc:348] T 8060ab0e7d594d6d8dd1d789844062a7 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.573701 25424 raft_consensus.cc:374] T 8060ab0e7d594d6d8dd1d789844062a7 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.573891 25424 raft_consensus.cc:729] T 8060ab0e7d594d6d8dd1d789844062a7 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.573964 25425 tablet_bootstrap.cc:654] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.574045 25424 consensus_queue.cc:260] T 8060ab0e7d594d6d8dd1d789844062a7 P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.574362 25424 ts_tablet_manager.cc:1398] T 8060ab0e7d594d6d8dd1d789844062a7 P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:56.575471 25419 tablet_bootstrap.cc:492] T 8060ab0e7d594d6d8dd1d789844062a7 P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:08:56.575507 25425 tablet_bootstrap.cc:492] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:08:56.575574 25419 ts_tablet_manager.cc:1367] T 8060ab0e7d594d6d8dd1d789844062a7 P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:08:56.575695 25425 ts_tablet_manager.cc:1367] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:08:56.576038 25419 raft_consensus.cc:348] T 8060ab0e7d594d6d8dd1d789844062a7 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.576143 25419 raft_consensus.cc:374] T 8060ab0e7d594d6d8dd1d789844062a7 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.576094 25425 raft_consensus.cc:348] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.576170 25419 raft_consensus.cc:729] T 8060ab0e7d594d6d8dd1d789844062a7 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.576179 25425 raft_consensus.cc:374] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.576208 25425 raft_consensus.cc:729] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.576300 25419 consensus_queue.cc:260] T 8060ab0e7d594d6d8dd1d789844062a7 P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.576310 25425 consensus_queue.cc:260] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.576547 25419 ts_tablet_manager.cc:1398] T 8060ab0e7d594d6d8dd1d789844062a7 P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:56.576547 25425 ts_tablet_manager.cc:1398] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:56.576694 25425 tablet_bootstrap.cc:492] T 430ccd0aed3948f9a91560ace381d83c P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:08:56.577549 25425 tablet_bootstrap.cc:654] T 430ccd0aed3948f9a91560ace381d83c P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.578615 25425 tablet_bootstrap.cc:492] T 430ccd0aed3948f9a91560ace381d83c P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:08:56.578694 25425 ts_tablet_manager.cc:1367] T 430ccd0aed3948f9a91560ace381d83c P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:56.579094 25425 raft_consensus.cc:348] T 430ccd0aed3948f9a91560ace381d83c P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.579193 25425 raft_consensus.cc:374] T 430ccd0aed3948f9a91560ace381d83c P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.579221 25425 raft_consensus.cc:729] T 430ccd0aed3948f9a91560ace381d83c P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.579355 25425 consensus_queue.cc:260] T 430ccd0aed3948f9a91560ace381d83c P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.579594 25425 ts_tablet_manager.cc:1398] T 430ccd0aed3948f9a91560ace381d83c P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:56.621857 25393 raft_consensus.cc:482] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:56.622030 25393 raft_consensus.cc:504] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.622442 25393 leader_election.cc:288] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:08:56.622851 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ae6674f0534b5eafc91d6579849ff0" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:08:56.622874 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ae6674f0534b5eafc91d6579849ff0" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79" is_pre_election: true
I20240501 14:08:56.622990 24502 raft_consensus.cc:2454] T c4ae6674f0534b5eafc91d6579849ff0 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 0.
I20240501 14:08:56.623021 25029 raft_consensus.cc:2454] T c4ae6674f0534b5eafc91d6579849ff0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 0.
I20240501 14:08:56.623323 24700 leader_election.cc:302] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:08:56.623502 25393 raft_consensus.cc:2781] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:56.623559 25393 raft_consensus.cc:482] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:56.623584 25393 raft_consensus.cc:3037] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.624394 25393 raft_consensus.cc:504] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.624706 25393 leader_election.cc:288] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 election: Requested vote from peers 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:08:56.625061 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ae6674f0534b5eafc91d6579849ff0" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:08:56.625056 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ae6674f0534b5eafc91d6579849ff0" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79"
I20240501 14:08:56.625169 24502 raft_consensus.cc:3037] T c4ae6674f0534b5eafc91d6579849ff0 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.625169 25029 raft_consensus.cc:3037] T c4ae6674f0534b5eafc91d6579849ff0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.626060 25029 raft_consensus.cc:2454] T c4ae6674f0534b5eafc91d6579849ff0 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 1.
I20240501 14:08:56.626060 24502 raft_consensus.cc:2454] T c4ae6674f0534b5eafc91d6579849ff0 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 1.
I20240501 14:08:56.626332 24700 leader_election.cc:302] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:08:56.626498 25393 raft_consensus.cc:2781] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:56.626582 25393 raft_consensus.cc:686] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 LEADER]: Becoming Leader. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Running, Role: LEADER
I20240501 14:08:56.626736 25393 consensus_queue.cc:237] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.628360 24357 catalog_manager.cc:5484] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67). New cstate: current_term: 1 leader_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:56.737318 25263 raft_consensus.cc:482] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:56.737457 25263 raft_consensus.cc:504] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.737890 25263 leader_election.cc:288] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345), c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971)
I20240501 14:08:56.738358 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8060ab0e7d594d6d8dd1d789844062a7" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3" is_pre_election: true
I20240501 14:08:56.738359 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8060ab0e7d594d6d8dd1d789844062a7" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6" is_pre_election: true
I20240501 14:08:56.738497 24634 raft_consensus.cc:2454] T 8060ab0e7d594d6d8dd1d789844062a7 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:08:56.738502 24898 raft_consensus.cc:2454] T 8060ab0e7d594d6d8dd1d789844062a7 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:08:56.738813 25098 leader_election.cc:302] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, c2691129fd82431bb867cb1307e009b6; no voters:
I20240501 14:08:56.738979 25263 raft_consensus.cc:2781] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:56.739041 25263 raft_consensus.cc:482] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:56.739074 25263 raft_consensus.cc:3037] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.739845 25263 raft_consensus.cc:504] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.740155 25263 leader_election.cc:288] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Requested vote from peers f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345), c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971)
I20240501 14:08:56.740482 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8060ab0e7d594d6d8dd1d789844062a7" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3"
I20240501 14:08:56.740515 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8060ab0e7d594d6d8dd1d789844062a7" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6"
I20240501 14:08:56.740593 24634 raft_consensus.cc:3037] T 8060ab0e7d594d6d8dd1d789844062a7 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.740613 24898 raft_consensus.cc:3037] T 8060ab0e7d594d6d8dd1d789844062a7 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.741379 24634 raft_consensus.cc:2454] T 8060ab0e7d594d6d8dd1d789844062a7 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:08:56.741380 24898 raft_consensus.cc:2454] T 8060ab0e7d594d6d8dd1d789844062a7 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:08:56.741658 25095 leader_election.cc:302] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:56.741852 25263 raft_consensus.cc:2781] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:56.741930 25263 raft_consensus.cc:686] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [term 1 LEADER]: Becoming Leader. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Running, Role: LEADER
I20240501 14:08:56.742059 25263 consensus_queue.cc:237] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:56.744043 24357 catalog_manager.cc:5484] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70). New cstate: current_term: 1 leader_uuid: "786f81b7c57f452e9b463f721fc3e865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:56.836609 25392 raft_consensus.cc:482] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:56.836755 25392 raft_consensus.cc:504] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.837785 25392 leader_election.cc:288] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:08:56.841403 24766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430ccd0aed3948f9a91560ace381d83c" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" is_pre_election: true
I20240501 14:08:56.841578 24766 raft_consensus.cc:2454] T 430ccd0aed3948f9a91560ace381d83c P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:08:56.841624 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430ccd0aed3948f9a91560ace381d83c" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:08:56.841779 24502 raft_consensus.cc:2454] T 430ccd0aed3948f9a91560ace381d83c P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:08:56.841948 24963 leader_election.cc:302] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:08:56.842181 25392 raft_consensus.cc:2781] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:56.842240 25392 raft_consensus.cc:482] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:56.842267 25392 raft_consensus.cc:3037] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.843055 25392 raft_consensus.cc:504] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.843395 25392 leader_election.cc:288] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Requested vote from peers 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:08:56.843730 24766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430ccd0aed3948f9a91560ace381d83c" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
I20240501 14:08:56.843753 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430ccd0aed3948f9a91560ace381d83c" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:08:56.843861 24502 raft_consensus.cc:3037] T 430ccd0aed3948f9a91560ace381d83c P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.843838 24766 raft_consensus.cc:3037] T 430ccd0aed3948f9a91560ace381d83c P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.844558 24502 raft_consensus.cc:2454] T 430ccd0aed3948f9a91560ace381d83c P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:08:56.844668 24766 raft_consensus.cc:2454] T 430ccd0aed3948f9a91560ace381d83c P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:08:56.844944 24964 leader_election.cc:302] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e375044d63649c097759c4b667eff79, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:08:56.848949 25392 raft_consensus.cc:2781] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:56.849040 25392 raft_consensus.cc:686] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [term 1 LEADER]: Becoming Leader. State: Replica: 7e375044d63649c097759c4b667eff79, State: Running, Role: LEADER
I20240501 14:08:56.849321 25392 consensus_queue.cc:237] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:56.851521 24357 catalog_manager.cc:5484] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e375044d63649c097759c4b667eff79 (127.23.191.69). New cstate: current_term: 1 leader_uuid: "7e375044d63649c097759c4b667eff79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } }
W20240501 14:08:56.969117 25440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:56.969360 25440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:56.972412 25440 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:08:57.061307 24766 raft_consensus.cc:1261] T 430ccd0aed3948f9a91560ace381d83c P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Refusing update from remote peer 7e375044d63649c097759c4b667eff79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:57.061393 24502 raft_consensus.cc:1261] T 430ccd0aed3948f9a91560ace381d83c P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Refusing update from remote peer 7e375044d63649c097759c4b667eff79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:57.061882 25252 consensus_queue.cc:1035] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:57.062062 25392 consensus_queue.cc:1035] T 430ccd0aed3948f9a91560ace381d83c P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:57.062387 24502 raft_consensus.cc:1261] T c4ae6674f0534b5eafc91d6579849ff0 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Refusing update from remote peer 319fbe37b2ce4434a8d2fe879ca920a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:57.063256 25245 consensus_queue.cc:1035] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:57.063792 25029 raft_consensus.cc:1261] T c4ae6674f0534b5eafc91d6579849ff0 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Refusing update from remote peer 319fbe37b2ce4434a8d2fe879ca920a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:57.064368 24898 raft_consensus.cc:1261] T 8060ab0e7d594d6d8dd1d789844062a7 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:57.064435 25393 consensus_queue.cc:1035] T c4ae6674f0534b5eafc91d6579849ff0 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:57.066529 25263 consensus_queue.cc:1035] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:57.074328 24634 raft_consensus.cc:1261] T 8060ab0e7d594d6d8dd1d789844062a7 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:57.075783 25263 consensus_queue.cc:1035] T 8060ab0e7d594d6d8dd1d789844062a7 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 110.157 ms
time per row: 0.0550784 ms
I20240501 14:08:57.125527 24358 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43068:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:57.131366 24878 tablet_service.cc:1467] Processing CreateTablet for tablet 5b6425c28a7a41fab2c271a06720e0c6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a8d3264eb73d448da44af1449f6ea7d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.131629 25143 tablet_service.cc:1467] Processing CreateTablet for tablet 5b6425c28a7a41fab2c271a06720e0c6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a8d3264eb73d448da44af1449f6ea7d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.131671 24877 tablet_service.cc:1467] Processing CreateTablet for tablet e9ff56af2d0040f4b3508807d1d4cb78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a8d3264eb73d448da44af1449f6ea7d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.131716 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b6425c28a7a41fab2c271a06720e0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.131913 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b6425c28a7a41fab2c271a06720e0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.132019 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ff56af2d0040f4b3508807d1d4cb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.133107 25142 tablet_service.cc:1467] Processing CreateTablet for tablet e9ff56af2d0040f4b3508807d1d4cb78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a8d3264eb73d448da44af1449f6ea7d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.133322 25142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ff56af2d0040f4b3508807d1d4cb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.133965 24613 tablet_service.cc:1467] Processing CreateTablet for tablet 5b6425c28a7a41fab2c271a06720e0c6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a8d3264eb73d448da44af1449f6ea7d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.134253 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b6425c28a7a41fab2c271a06720e0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.135092 25504 tablet_bootstrap.cc:492] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:08:57.135823 25504 tablet_bootstrap.cc:654] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.136024 25505 tablet_bootstrap.cc:492] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:08:57.136615 24482 tablet_service.cc:1467] Processing CreateTablet for tablet 90c5d177fbdc4be5b77fff955af6532b (DEFAULT_TABLE table=rebalance_test_table_3 [id=a8d3264eb73d448da44af1449f6ea7d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.136937 24482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c5d177fbdc4be5b77fff955af6532b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.137040 25505 tablet_bootstrap.cc:654] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.137054 24613 tablet_service.cc:1467] Processing CreateTablet for tablet e9ff56af2d0040f4b3508807d1d4cb78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a8d3264eb73d448da44af1449f6ea7d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.137272 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ff56af2d0040f4b3508807d1d4cb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.137281 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 90c5d177fbdc4be5b77fff955af6532b (DEFAULT_TABLE table=rebalance_test_table_3 [id=a8d3264eb73d448da44af1449f6ea7d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.137521 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c5d177fbdc4be5b77fff955af6532b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.138724 25508 tablet_bootstrap.cc:492] T 90c5d177fbdc4be5b77fff955af6532b P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:08:57.139179 25010 tablet_service.cc:1467] Processing CreateTablet for tablet 90c5d177fbdc4be5b77fff955af6532b (DEFAULT_TABLE table=rebalance_test_table_3 [id=a8d3264eb73d448da44af1449f6ea7d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.139485 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c5d177fbdc4be5b77fff955af6532b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.139533 25509 tablet_bootstrap.cc:492] T 90c5d177fbdc4be5b77fff955af6532b P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:08:57.139575 25508 tablet_bootstrap.cc:654] T 90c5d177fbdc4be5b77fff955af6532b P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.140405 25509 tablet_bootstrap.cc:654] T 90c5d177fbdc4be5b77fff955af6532b P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.141368 25503 tablet_bootstrap.cc:492] T e9ff56af2d0040f4b3508807d1d4cb78 P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:08:57.141875 25504 tablet_bootstrap.cc:492] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:08:57.141973 25504 ts_tablet_manager.cc:1367] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:08:57.142154 25503 tablet_bootstrap.cc:654] T e9ff56af2d0040f4b3508807d1d4cb78 P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.142480 25512 tablet_bootstrap.cc:492] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:08:57.142457 25504 raft_consensus.cc:348] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.142565 25504 raft_consensus.cc:374] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.142599 25504 raft_consensus.cc:729] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.142724 25504 consensus_queue.cc:260] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.143010 25504 ts_tablet_manager.cc:1398] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:57.143174 25504 tablet_bootstrap.cc:492] T 5b6425c28a7a41fab2c271a06720e0c6 P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:08:57.143330 25509 tablet_bootstrap.cc:492] T 90c5d177fbdc4be5b77fff955af6532b P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:08:57.143435 25509 ts_tablet_manager.cc:1367] T 90c5d177fbdc4be5b77fff955af6532b P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:57.143560 25508 tablet_bootstrap.cc:492] T 90c5d177fbdc4be5b77fff955af6532b P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:08:57.143638 25508 ts_tablet_manager.cc:1367] T 90c5d177fbdc4be5b77fff955af6532b P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:08:57.143894 25509 raft_consensus.cc:348] T 90c5d177fbdc4be5b77fff955af6532b P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:57.143996 25509 raft_consensus.cc:374] T 90c5d177fbdc4be5b77fff955af6532b P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.144029 25509 raft_consensus.cc:729] T 90c5d177fbdc4be5b77fff955af6532b P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.144109 25504 tablet_bootstrap.cc:654] T 5b6425c28a7a41fab2c271a06720e0c6 P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.144145 25509 consensus_queue.cc:260] T 90c5d177fbdc4be5b77fff955af6532b P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:57.144316 25508 raft_consensus.cc:348] T 90c5d177fbdc4be5b77fff955af6532b P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:57.144412 25509 ts_tablet_manager.cc:1398] T 90c5d177fbdc4be5b77fff955af6532b P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:57.144420 25508 raft_consensus.cc:374] T 90c5d177fbdc4be5b77fff955af6532b P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.144454 25508 raft_consensus.cc:729] T 90c5d177fbdc4be5b77fff955af6532b P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.144604 25508 consensus_queue.cc:260] T 90c5d177fbdc4be5b77fff955af6532b P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:57.145512 25508 ts_tablet_manager.cc:1398] T 90c5d177fbdc4be5b77fff955af6532b P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:57.145952 25503 tablet_bootstrap.cc:492] T e9ff56af2d0040f4b3508807d1d4cb78 P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:08:57.146041 25503 ts_tablet_manager.cc:1367] T e9ff56af2d0040f4b3508807d1d4cb78 P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:08:57.146472 25503 raft_consensus.cc:348] T e9ff56af2d0040f4b3508807d1d4cb78 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.146585 25503 raft_consensus.cc:374] T e9ff56af2d0040f4b3508807d1d4cb78 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.146603 25505 tablet_bootstrap.cc:492] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:08:57.146620 25503 raft_consensus.cc:729] T e9ff56af2d0040f4b3508807d1d4cb78 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.146706 25505 ts_tablet_manager.cc:1367] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:08:57.146744 25503 consensus_queue.cc:260] T e9ff56af2d0040f4b3508807d1d4cb78 P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.147019 25503 ts_tablet_manager.cc:1398] T e9ff56af2d0040f4b3508807d1d4cb78 P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:57.147186 25503 tablet_bootstrap.cc:492] T 5b6425c28a7a41fab2c271a06720e0c6 P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:08:57.147132 25505 raft_consensus.cc:348] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:57.147236 25505 raft_consensus.cc:374] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.147269 25505 raft_consensus.cc:729] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.147382 25505 consensus_queue.cc:260] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:57.147636 25505 ts_tablet_manager.cc:1398] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:57.147686 25512 tablet_bootstrap.cc:654] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.147818 25505 tablet_bootstrap.cc:492] T e9ff56af2d0040f4b3508807d1d4cb78 P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:08:57.148052 25503 tablet_bootstrap.cc:654] T 5b6425c28a7a41fab2c271a06720e0c6 P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.148645 25505 tablet_bootstrap.cc:654] T e9ff56af2d0040f4b3508807d1d4cb78 P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.149075 25512 tablet_bootstrap.cc:492] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:08:57.149155 25512 ts_tablet_manager.cc:1367] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:08:57.149585 25512 raft_consensus.cc:348] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:57.149690 25505 tablet_bootstrap.cc:492] T e9ff56af2d0040f4b3508807d1d4cb78 P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:08:57.149691 25512 raft_consensus.cc:374] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.149724 25512 raft_consensus.cc:729] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.149771 25505 ts_tablet_manager.cc:1367] T e9ff56af2d0040f4b3508807d1d4cb78 P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:57.149853 25512 consensus_queue.cc:260] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:57.150146 25512 ts_tablet_manager.cc:1398] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:57.150178 25505 raft_consensus.cc:348] T e9ff56af2d0040f4b3508807d1d4cb78 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.150280 25505 raft_consensus.cc:374] T e9ff56af2d0040f4b3508807d1d4cb78 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.150311 25505 raft_consensus.cc:729] T e9ff56af2d0040f4b3508807d1d4cb78 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.150416 25505 consensus_queue.cc:260] T e9ff56af2d0040f4b3508807d1d4cb78 P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.150686 25505 ts_tablet_manager.cc:1398] T e9ff56af2d0040f4b3508807d1d4cb78 P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:57.150844 25504 tablet_bootstrap.cc:492] T 5b6425c28a7a41fab2c271a06720e0c6 P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:08:57.150930 25504 ts_tablet_manager.cc:1367] T 5b6425c28a7a41fab2c271a06720e0c6 P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:08:57.151486 25504 raft_consensus.cc:348] T 5b6425c28a7a41fab2c271a06720e0c6 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:57.151610 25504 raft_consensus.cc:374] T 5b6425c28a7a41fab2c271a06720e0c6 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.151928 25504 raft_consensus.cc:729] T 5b6425c28a7a41fab2c271a06720e0c6 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.152076 25504 consensus_queue.cc:260] T 5b6425c28a7a41fab2c271a06720e0c6 P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:57.152572 25504 ts_tablet_manager.cc:1398] T 5b6425c28a7a41fab2c271a06720e0c6 P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:57.153982 25503 tablet_bootstrap.cc:492] T 5b6425c28a7a41fab2c271a06720e0c6 P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:08:57.154086 25503 ts_tablet_manager.cc:1367] T 5b6425c28a7a41fab2c271a06720e0c6 P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:08:57.154517 25503 raft_consensus.cc:348] T 5b6425c28a7a41fab2c271a06720e0c6 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:57.154626 25503 raft_consensus.cc:374] T 5b6425c28a7a41fab2c271a06720e0c6 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.154668 25503 raft_consensus.cc:729] T 5b6425c28a7a41fab2c271a06720e0c6 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.154793 25503 consensus_queue.cc:260] T 5b6425c28a7a41fab2c271a06720e0c6 P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:57.155027 25503 ts_tablet_manager.cc:1398] T 5b6425c28a7a41fab2c271a06720e0c6 P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:57.176038 25263 raft_consensus.cc:482] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:57.176182 25263 raft_consensus.cc:504] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.176589 25263 leader_election.cc:288] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:08:57.177021 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ff56af2d0040f4b3508807d1d4cb78" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6" is_pre_election: true
I20240501 14:08:57.177042 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ff56af2d0040f4b3508807d1d4cb78" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3" is_pre_election: true
I20240501 14:08:57.177165 24898 raft_consensus.cc:2454] T e9ff56af2d0040f4b3508807d1d4cb78 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:08:57.177181 24634 raft_consensus.cc:2454] T e9ff56af2d0040f4b3508807d1d4cb78 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:08:57.177462 25098 leader_election.cc:302] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, c2691129fd82431bb867cb1307e009b6; no voters:
I20240501 14:08:57.177632 25263 raft_consensus.cc:2781] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:57.177698 25263 raft_consensus.cc:482] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:57.177728 25263 raft_consensus.cc:3037] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:57.227360 25263 raft_consensus.cc:504] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.227896 25263 leader_election.cc:288] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Requested vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:08:57.228483 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ff56af2d0040f4b3508807d1d4cb78" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3"
I20240501 14:08:57.228519 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ff56af2d0040f4b3508807d1d4cb78" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6"
I20240501 14:08:57.228639 24634 raft_consensus.cc:3037] T e9ff56af2d0040f4b3508807d1d4cb78 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:57.228648 24898 raft_consensus.cc:3037] T e9ff56af2d0040f4b3508807d1d4cb78 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:57.229673 24898 raft_consensus.cc:2454] T e9ff56af2d0040f4b3508807d1d4cb78 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:08:57.229674 24634 raft_consensus.cc:2454] T e9ff56af2d0040f4b3508807d1d4cb78 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:08:57.230202 25095 leader_election.cc:302] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:57.230454 25263 raft_consensus.cc:2781] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:57.230553 25263 raft_consensus.cc:686] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [term 1 LEADER]: Becoming Leader. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Running, Role: LEADER
I20240501 14:08:57.230687 25263 consensus_queue.cc:237] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.232722 24353 catalog_manager.cc:5484] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70). New cstate: current_term: 1 leader_uuid: "786f81b7c57f452e9b463f721fc3e865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:57.233567 25490 raft_consensus.cc:482] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:57.233671 25490 raft_consensus.cc:504] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:57.234910 25490 leader_election.cc:288] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343)
I20240501 14:08:57.238492 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6425c28a7a41fab2c271a06720e0c6" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865" is_pre_election: true
I20240501 14:08:57.238667 25163 raft_consensus.cc:2454] T 5b6425c28a7a41fab2c271a06720e0c6 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 0.
I20240501 14:08:57.239034 24569 leader_election.cc:302] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:57.239171 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6425c28a7a41fab2c271a06720e0c6" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6" is_pre_election: true
I20240501 14:08:57.239276 25490 raft_consensus.cc:2781] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:57.239317 24898 raft_consensus.cc:2454] T 5b6425c28a7a41fab2c271a06720e0c6 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 0.
I20240501 14:08:57.239342 25490 raft_consensus.cc:482] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:57.239379 25490 raft_consensus.cc:3037] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:57.329033 25490 raft_consensus.cc:504] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:57.329506 25490 leader_election.cc:288] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 election: Requested vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343)
I20240501 14:08:57.329988 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6425c28a7a41fab2c271a06720e0c6" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865"
I20240501 14:08:57.330019 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6425c28a7a41fab2c271a06720e0c6" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6"
I20240501 14:08:57.330137 24898 raft_consensus.cc:3037] T 5b6425c28a7a41fab2c271a06720e0c6 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:57.330137 25163 raft_consensus.cc:3037] T 5b6425c28a7a41fab2c271a06720e0c6 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:57.331110 25163 raft_consensus.cc:2454] T 5b6425c28a7a41fab2c271a06720e0c6 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 1.
I20240501 14:08:57.331110 24898 raft_consensus.cc:2454] T 5b6425c28a7a41fab2c271a06720e0c6 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 1.
I20240501 14:08:57.331486 24570 leader_election.cc:302] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2691129fd82431bb867cb1307e009b6, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:57.331701 25490 raft_consensus.cc:2781] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:57.331964 25490 raft_consensus.cc:686] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 LEADER]: Becoming Leader. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Running, Role: LEADER
I20240501 14:08:57.332304 25490 consensus_queue.cc:237] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:57.333681 25499 raft_consensus.cc:482] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:57.333813 25499 raft_consensus.cc:504] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:57.334295 25499 leader_election.cc:288] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:08:57.334699 24766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c5d177fbdc4be5b77fff955af6532b" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" is_pre_election: true
I20240501 14:08:57.334817 24766 raft_consensus.cc:2454] T 90c5d177fbdc4be5b77fff955af6532b P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:08:57.335067 24963 leader_election.cc:302] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:08:57.335317 24353 catalog_manager.cc:5484] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66). New cstate: current_term: 1 leader_uuid: "f68136de04c94e5cb4a5f99881f02ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:57.335577 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c5d177fbdc4be5b77fff955af6532b" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:08:57.335731 24501 raft_consensus.cc:2454] T 90c5d177fbdc4be5b77fff955af6532b P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:08:57.335937 25499 raft_consensus.cc:2781] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:57.335997 25499 raft_consensus.cc:482] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:57.336032 25499 raft_consensus.cc:3037] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:57.427029 25499 raft_consensus.cc:504] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:57.427505 25499 leader_election.cc:288] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Requested vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:08:57.428124 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c5d177fbdc4be5b77fff955af6532b" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:08:57.428124 24766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c5d177fbdc4be5b77fff955af6532b" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
I20240501 14:08:57.428268 24501 raft_consensus.cc:3037] T 90c5d177fbdc4be5b77fff955af6532b P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:57.428268 24766 raft_consensus.cc:3037] T 90c5d177fbdc4be5b77fff955af6532b P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:57.429296 24766 raft_consensus.cc:2454] T 90c5d177fbdc4be5b77fff955af6532b P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:08:57.429301 24501 raft_consensus.cc:2454] T 90c5d177fbdc4be5b77fff955af6532b P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:08:57.429621 24964 leader_election.cc:302] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e375044d63649c097759c4b667eff79, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:08:57.429811 25499 raft_consensus.cc:2781] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:57.429903 25499 raft_consensus.cc:686] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [term 1 LEADER]: Becoming Leader. State: Replica: 7e375044d63649c097759c4b667eff79, State: Running, Role: LEADER
I20240501 14:08:57.430042 25499 consensus_queue.cc:237] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:57.431836 24353 catalog_manager.cc:5484] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e375044d63649c097759c4b667eff79 (127.23.191.69). New cstate: current_term: 1 leader_uuid: "7e375044d63649c097759c4b667eff79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: UNKNOWN } } }
W20240501 14:08:57.548094 25525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:57.548348 25525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:57.551466 25525 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 14:08:57.608131 25263 consensus_queue.cc:1035] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:57.612720 25247 consensus_queue.cc:1035] T e9ff56af2d0040f4b3508807d1d4cb78 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:57.642952 25163 raft_consensus.cc:1261] T 5b6425c28a7a41fab2c271a06720e0c6 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Refusing update from remote peer f68136de04c94e5cb4a5f99881f02ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:57.643812 25490 consensus_queue.cc:1035] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:57.644904 24766 raft_consensus.cc:1261] T 90c5d177fbdc4be5b77fff955af6532b P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Refusing update from remote peer 7e375044d63649c097759c4b667eff79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:57.646598 24898 raft_consensus.cc:1261] T 5b6425c28a7a41fab2c271a06720e0c6 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Refusing update from remote peer f68136de04c94e5cb4a5f99881f02ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:57.646981 25490 consensus_queue.cc:1035] T 5b6425c28a7a41fab2c271a06720e0c6 P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:57.651865 25499 consensus_queue.cc:1035] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:57.666908 24501 raft_consensus.cc:1261] T 90c5d177fbdc4be5b77fff955af6532b P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Refusing update from remote peer 7e375044d63649c097759c4b667eff79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:57.667454 25499 consensus_queue.cc:1035] T 90c5d177fbdc4be5b77fff955af6532b P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 120.966 ms
time per row: 0.0604831 ms
I20240501 14:08:57.714673 24357 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43068:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:57.720930 24482 tablet_service.cc:1467] Processing CreateTablet for tablet d755aa2b9cd241ec8acede8f8fd46533 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b01a2db840b5481eb2a19085c70bf2ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.721863 24481 tablet_service.cc:1467] Processing CreateTablet for tablet db3f5aae662b42b49a306f978985fd39 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b01a2db840b5481eb2a19085c70bf2ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.722631 24746 tablet_service.cc:1467] Processing CreateTablet for tablet db3f5aae662b42b49a306f978985fd39 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b01a2db840b5481eb2a19085c70bf2ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.722613 24613 tablet_service.cc:1467] Processing CreateTablet for tablet e2affc5e80214d0fae061d7b2bb4f013 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b01a2db840b5481eb2a19085c70bf2ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.722882 24746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3f5aae662b42b49a306f978985fd39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.722885 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2affc5e80214d0fae061d7b2bb4f013. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.724704 25010 tablet_service.cc:1467] Processing CreateTablet for tablet d755aa2b9cd241ec8acede8f8fd46533 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b01a2db840b5481eb2a19085c70bf2ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.725082 25009 tablet_service.cc:1467] Processing CreateTablet for tablet db3f5aae662b42b49a306f978985fd39 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b01a2db840b5481eb2a19085c70bf2ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.725777 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d755aa2b9cd241ec8acede8f8fd46533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.728528 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3f5aae662b42b49a306f978985fd39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.728963 24878 tablet_service.cc:1467] Processing CreateTablet for tablet e2affc5e80214d0fae061d7b2bb4f013 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b01a2db840b5481eb2a19085c70bf2ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.729131 24745 tablet_service.cc:1467] Processing CreateTablet for tablet d755aa2b9cd241ec8acede8f8fd46533 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b01a2db840b5481eb2a19085c70bf2ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.729249 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2affc5e80214d0fae061d7b2bb4f013. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.729357 24482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d755aa2b9cd241ec8acede8f8fd46533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.729395 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d755aa2b9cd241ec8acede8f8fd46533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.729409 25596 tablet_bootstrap.cc:492] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:08:57.730315 25596 tablet_bootstrap.cc:654] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.731351 25596 tablet_bootstrap.cc:492] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:08:57.731429 25596 ts_tablet_manager.cc:1367] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:57.728816 25143 tablet_service.cc:1467] Processing CreateTablet for tablet e2affc5e80214d0fae061d7b2bb4f013 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b01a2db840b5481eb2a19085c70bf2ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:57.731906 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2affc5e80214d0fae061d7b2bb4f013. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.731885 25596 raft_consensus.cc:348] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.731994 25596 raft_consensus.cc:374] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.732026 25596 raft_consensus.cc:729] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.732147 25596 consensus_queue.cc:260] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.732232 24481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3f5aae662b42b49a306f978985fd39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.732404 25596 ts_tablet_manager.cc:1398] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:57.733991 25601 tablet_bootstrap.cc:492] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:08:57.734189 25598 tablet_bootstrap.cc:492] T d755aa2b9cd241ec8acede8f8fd46533 P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:08:57.734428 25599 tablet_bootstrap.cc:492] T e2affc5e80214d0fae061d7b2bb4f013 P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:08:57.734942 25601 tablet_bootstrap.cc:654] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.735900 25599 tablet_bootstrap.cc:654] T e2affc5e80214d0fae061d7b2bb4f013 P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.736699 25596 tablet_bootstrap.cc:492] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:08:57.737134 25598 tablet_bootstrap.cc:654] T d755aa2b9cd241ec8acede8f8fd46533 P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.737314 25599 tablet_bootstrap.cc:492] T e2affc5e80214d0fae061d7b2bb4f013 P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:08:57.737597 25599 ts_tablet_manager.cc:1367] T e2affc5e80214d0fae061d7b2bb4f013 P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:57.737622 25596 tablet_bootstrap.cc:654] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.738020 25599 raft_consensus.cc:348] T e2affc5e80214d0fae061d7b2bb4f013 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.738124 25599 raft_consensus.cc:374] T e2affc5e80214d0fae061d7b2bb4f013 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.738152 25599 raft_consensus.cc:729] T e2affc5e80214d0fae061d7b2bb4f013 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.738328 25599 consensus_queue.cc:260] T e2affc5e80214d0fae061d7b2bb4f013 P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.738463 25600 tablet_bootstrap.cc:492] T d755aa2b9cd241ec8acede8f8fd46533 P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:08:57.738822 25599 ts_tablet_manager.cc:1398] T e2affc5e80214d0fae061d7b2bb4f013 P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:57.740195 25600 tablet_bootstrap.cc:654] T d755aa2b9cd241ec8acede8f8fd46533 P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.740752 25596 tablet_bootstrap.cc:492] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:08:57.740867 25596 ts_tablet_manager.cc:1367] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:57.741384 25596 raft_consensus.cc:348] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.741495 25596 raft_consensus.cc:374] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.741562 25598 tablet_bootstrap.cc:492] T d755aa2b9cd241ec8acede8f8fd46533 P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:08:57.741600 25600 tablet_bootstrap.cc:492] T d755aa2b9cd241ec8acede8f8fd46533 P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:08:57.741588 25596 raft_consensus.cc:729] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.741672 25600 ts_tablet_manager.cc:1367] T d755aa2b9cd241ec8acede8f8fd46533 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:57.741773 25598 ts_tablet_manager.cc:1367] T d755aa2b9cd241ec8acede8f8fd46533 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:08:57.741775 25596 consensus_queue.cc:260] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.741936 25589 raft_consensus.cc:482] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:57.742007 25589 raft_consensus.cc:504] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.742188 25598 raft_consensus.cc:348] T d755aa2b9cd241ec8acede8f8fd46533 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.742278 25598 raft_consensus.cc:374] T d755aa2b9cd241ec8acede8f8fd46533 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.742308 25598 raft_consensus.cc:729] T d755aa2b9cd241ec8acede8f8fd46533 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.742369 25589 leader_election.cc:288] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:08:57.742491 25596 ts_tablet_manager.cc:1398] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:57.742449 25598 consensus_queue.cc:260] T d755aa2b9cd241ec8acede8f8fd46533 P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.742714 25598 ts_tablet_manager.cc:1398] T d755aa2b9cd241ec8acede8f8fd46533 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:57.743073 25598 tablet_bootstrap.cc:492] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:08:57.743098 24766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3f5aae662b42b49a306f978985fd39" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" is_pre_election: true
I20240501 14:08:57.743501 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3f5aae662b42b49a306f978985fd39" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:08:57.743990 25598 tablet_bootstrap.cc:654] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.744330 25600 raft_consensus.cc:348] T d755aa2b9cd241ec8acede8f8fd46533 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.744464 25600 raft_consensus.cc:374] T d755aa2b9cd241ec8acede8f8fd46533 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.744520 25600 raft_consensus.cc:729] T d755aa2b9cd241ec8acede8f8fd46533 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.744657 25600 consensus_queue.cc:260] T d755aa2b9cd241ec8acede8f8fd46533 P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.745141 25600 ts_tablet_manager.cc:1398] T d755aa2b9cd241ec8acede8f8fd46533 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:08:57.745344 25600 tablet_bootstrap.cc:492] T db3f5aae662b42b49a306f978985fd39 P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:08:57.745424 25598 tablet_bootstrap.cc:492] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:08:57.745543 25598 ts_tablet_manager.cc:1367] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:57.745980 25598 raft_consensus.cc:348] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.746109 25598 raft_consensus.cc:374] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.746152 25598 raft_consensus.cc:729] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.746263 25600 tablet_bootstrap.cc:654] T db3f5aae662b42b49a306f978985fd39 P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.746281 25598 consensus_queue.cc:260] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.746543 25598 ts_tablet_manager.cc:1398] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:57.747416 25600 tablet_bootstrap.cc:492] T db3f5aae662b42b49a306f978985fd39 P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:08:57.747493 25601 tablet_bootstrap.cc:492] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:08:57.747543 25600 ts_tablet_manager.cc:1367] T db3f5aae662b42b49a306f978985fd39 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:57.747596 25601 ts_tablet_manager.cc:1367] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:08:57.748047 25601 raft_consensus.cc:348] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.748167 25601 raft_consensus.cc:374] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.748210 25601 raft_consensus.cc:729] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.748260 25600 raft_consensus.cc:348] T db3f5aae662b42b49a306f978985fd39 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.748328 25601 consensus_queue.cc:260] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.748353 25600 raft_consensus.cc:374] T db3f5aae662b42b49a306f978985fd39 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.748471 25600 raft_consensus.cc:729] T db3f5aae662b42b49a306f978985fd39 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.748585 25601 ts_tablet_manager.cc:1398] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:57.748612 25600 consensus_queue.cc:260] T db3f5aae662b42b49a306f978985fd39 P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
W20240501 14:08:57.744053 24964 leader_election.cc:341] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:08:57.748823 24963 leader_election.cc:341] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:08:57.749111 24963 leader_election.cc:302] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e375044d63649c097759c4b667eff79; no voters: 319fbe37b2ce4434a8d2fe879ca920a7, fd8ec7cff3a147e5adfc4eb329a23895
I20240501 14:08:57.749374 25589 raft_consensus.cc:2726] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:08:57.749692 25600 ts_tablet_manager.cc:1398] T db3f5aae662b42b49a306f978985fd39 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:57.750058 25595 tablet_bootstrap.cc:492] T e2affc5e80214d0fae061d7b2bb4f013 P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:08:57.750782 25595 tablet_bootstrap.cc:654] T e2affc5e80214d0fae061d7b2bb4f013 P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.751857 25595 tablet_bootstrap.cc:492] T e2affc5e80214d0fae061d7b2bb4f013 P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:08:57.751953 25595 ts_tablet_manager.cc:1367] T e2affc5e80214d0fae061d7b2bb4f013 P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:57.752421 25595 raft_consensus.cc:348] T e2affc5e80214d0fae061d7b2bb4f013 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.752557 25595 raft_consensus.cc:374] T e2affc5e80214d0fae061d7b2bb4f013 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.752609 25595 raft_consensus.cc:729] T e2affc5e80214d0fae061d7b2bb4f013 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.752959 25595 consensus_queue.cc:260] T e2affc5e80214d0fae061d7b2bb4f013 P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.753242 25595 ts_tablet_manager.cc:1398] T e2affc5e80214d0fae061d7b2bb4f013 P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:57.868471 25589 raft_consensus.cc:482] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:57.868625 25589 raft_consensus.cc:504] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.869089 25589 leader_election.cc:288] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:08:57.869524 24766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d755aa2b9cd241ec8acede8f8fd46533" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" is_pre_election: true
I20240501 14:08:57.869534 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d755aa2b9cd241ec8acede8f8fd46533" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:08:57.869666 24766 raft_consensus.cc:2454] T d755aa2b9cd241ec8acede8f8fd46533 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:08:57.869683 24501 raft_consensus.cc:2454] T d755aa2b9cd241ec8acede8f8fd46533 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:08:57.869982 24963 leader_election.cc:302] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:08:57.870143 25589 raft_consensus.cc:2781] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:57.870203 25589 raft_consensus.cc:482] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:57.870235 25589 raft_consensus.cc:3037] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:57.932941 25589 raft_consensus.cc:504] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.933426 25589 leader_election.cc:288] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Requested vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:08:57.933815 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d755aa2b9cd241ec8acede8f8fd46533" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:08:57.933971 24501 raft_consensus.cc:3037] T d755aa2b9cd241ec8acede8f8fd46533 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:57.933920 24766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d755aa2b9cd241ec8acede8f8fd46533" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
I20240501 14:08:57.934062 24766 raft_consensus.cc:3037] T d755aa2b9cd241ec8acede8f8fd46533 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:57.934937 24501 raft_consensus.cc:2454] T d755aa2b9cd241ec8acede8f8fd46533 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:08:57.935015 24766 raft_consensus.cc:2454] T d755aa2b9cd241ec8acede8f8fd46533 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:08:57.935297 24964 leader_election.cc:302] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e375044d63649c097759c4b667eff79, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:08:57.935532 25589 raft_consensus.cc:2781] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:57.935660 25589 raft_consensus.cc:686] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [term 1 LEADER]: Becoming Leader. State: Replica: 7e375044d63649c097759c4b667eff79, State: Running, Role: LEADER
I20240501 14:08:57.935809 25589 consensus_queue.cc:237] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:57.937709 24353 catalog_manager.cc:5484] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e375044d63649c097759c4b667eff79 (127.23.191.69). New cstate: current_term: 1 leader_uuid: "7e375044d63649c097759c4b667eff79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:57.976487 25247 raft_consensus.cc:482] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:57.976613 25247 raft_consensus.cc:504] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:57.977049 25247 leader_election.cc:288] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:08:57.977470 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2affc5e80214d0fae061d7b2bb4f013" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6" is_pre_election: true
I20240501 14:08:57.977504 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2affc5e80214d0fae061d7b2bb4f013" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3" is_pre_election: true
I20240501 14:08:57.977619 24897 raft_consensus.cc:2454] T e2affc5e80214d0fae061d7b2bb4f013 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:08:57.977656 24634 raft_consensus.cc:2454] T e2affc5e80214d0fae061d7b2bb4f013 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:08:57.977916 25098 leader_election.cc:302] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, c2691129fd82431bb867cb1307e009b6; no voters:
I20240501 14:08:57.978066 25247 raft_consensus.cc:2781] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:57.978137 25247 raft_consensus.cc:482] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:57.978188 25247 raft_consensus.cc:3037] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.035183 25247 raft_consensus.cc:504] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:58.035665 25247 leader_election.cc:288] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Requested vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:08:58.036142 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2affc5e80214d0fae061d7b2bb4f013" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3"
I20240501 14:08:58.036130 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2affc5e80214d0fae061d7b2bb4f013" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6"
I20240501 14:08:58.036278 24634 raft_consensus.cc:3037] T e2affc5e80214d0fae061d7b2bb4f013 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.036281 24897 raft_consensus.cc:3037] T e2affc5e80214d0fae061d7b2bb4f013 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.037248 24634 raft_consensus.cc:2454] T e2affc5e80214d0fae061d7b2bb4f013 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:08:58.037331 24897 raft_consensus.cc:2454] T e2affc5e80214d0fae061d7b2bb4f013 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:08:58.037571 25095 leader_election.cc:302] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:58.037739 25247 raft_consensus.cc:2781] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:58.037801 25247 raft_consensus.cc:686] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [term 1 LEADER]: Becoming Leader. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Running, Role: LEADER
I20240501 14:08:58.037927 25247 consensus_queue.cc:237] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:58.039937 24353 catalog_manager.cc:5484] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70). New cstate: current_term: 1 leader_uuid: "786f81b7c57f452e9b463f721fc3e865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:58.046779 25573 raft_consensus.cc:482] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:58.046900 25573 raft_consensus.cc:504] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:58.047276 25573 leader_election.cc:288] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635)
I20240501 14:08:58.047667 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3f5aae662b42b49a306f978985fd39" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:08:58.047691 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3f5aae662b42b49a306f978985fd39" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79" is_pre_election: true
I20240501 14:08:58.047799 24501 raft_consensus.cc:2454] T db3f5aae662b42b49a306f978985fd39 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 0.
I20240501 14:08:58.047852 25029 raft_consensus.cc:2454] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 0.
I20240501 14:08:58.048156 24699 leader_election.cc:302] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:08:58.048355 25573 raft_consensus.cc:2781] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:58.048425 25573 raft_consensus.cc:482] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:58.048470 25573 raft_consensus.cc:3037] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.132804 25573 raft_consensus.cc:504] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:58.133237 25573 leader_election.cc:288] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 election: Requested vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635)
I20240501 14:08:58.133644 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3f5aae662b42b49a306f978985fd39" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:08:58.133644 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3f5aae662b42b49a306f978985fd39" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79"
I20240501 14:08:58.133783 24501 raft_consensus.cc:3037] T db3f5aae662b42b49a306f978985fd39 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.133783 25029 raft_consensus.cc:3037] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.134721 25029 raft_consensus.cc:2454] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 1.
I20240501 14:08:58.134721 24501 raft_consensus.cc:2454] T db3f5aae662b42b49a306f978985fd39 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 1.
I20240501 14:08:58.135037 24700 leader_election.cc:302] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:08:58.135241 25573 raft_consensus.cc:2781] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:58.135340 25573 raft_consensus.cc:686] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 LEADER]: Becoming Leader. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Running, Role: LEADER
I20240501 14:08:58.135476 25573 consensus_queue.cc:237] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:08:58.137218 24353 catalog_manager.cc:5484] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67). New cstate: current_term: 1 leader_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: UNKNOWN } } }
W20240501 14:08:58.294720 25613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:58.294989 25613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:58.298178 25613 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 14:08:58.374826 24766 raft_consensus.cc:1261] T d755aa2b9cd241ec8acede8f8fd46533 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Refusing update from remote peer 7e375044d63649c097759c4b667eff79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:58.375269 24501 raft_consensus.cc:1261] T d755aa2b9cd241ec8acede8f8fd46533 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Refusing update from remote peer 7e375044d63649c097759c4b667eff79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:58.375537 25589 consensus_queue.cc:1035] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:58.375880 25589 consensus_queue.cc:1035] T d755aa2b9cd241ec8acede8f8fd46533 P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:58.380545 24502 raft_consensus.cc:1261] T db3f5aae662b42b49a306f978985fd39 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Refusing update from remote peer 319fbe37b2ce4434a8d2fe879ca920a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:58.380553 25029 raft_consensus.cc:1261] T db3f5aae662b42b49a306f978985fd39 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Refusing update from remote peer 319fbe37b2ce4434a8d2fe879ca920a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:58.380942 25393 consensus_queue.cc:1035] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:58.381076 25573 consensus_queue.cc:1035] T db3f5aae662b42b49a306f978985fd39 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:58.387702 24634 raft_consensus.cc:1261] T e2affc5e80214d0fae061d7b2bb4f013 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:58.387943 24897 raft_consensus.cc:1261] T e2affc5e80214d0fae061d7b2bb4f013 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:58.388228 25263 consensus_queue.cc:1035] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:58.388502 25247 consensus_queue.cc:1035] T e2affc5e80214d0fae061d7b2bb4f013 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.259 ms
time per row: 0.0556297 ms
I20240501 14:08:58.454921 24357 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43068:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:58.460553 24613 tablet_service.cc:1467] Processing CreateTablet for tablet 44d3c8472d1543938d1bbbfd85b79447 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8384a6fd4134f569415b3c47c642ec9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:58.460598 25143 tablet_service.cc:1467] Processing CreateTablet for tablet 44d3c8472d1543938d1bbbfd85b79447 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8384a6fd4134f569415b3c47c642ec9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:58.460927 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d3c8472d1543938d1bbbfd85b79447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:58.460956 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d3c8472d1543938d1bbbfd85b79447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:58.461364 25009 tablet_service.cc:1467] Processing CreateTablet for tablet 94b711edc7694340a0c4a68ef178d8fe (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8384a6fd4134f569415b3c47c642ec9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:58.461613 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94b711edc7694340a0c4a68ef178d8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:58.462659 25684 tablet_bootstrap.cc:492] T 44d3c8472d1543938d1bbbfd85b79447 P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:08:58.462910 24746 tablet_service.cc:1467] Processing CreateTablet for tablet 94b711edc7694340a0c4a68ef178d8fe (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8384a6fd4134f569415b3c47c642ec9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:58.463217 24746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94b711edc7694340a0c4a68ef178d8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:58.463589 25684 tablet_bootstrap.cc:654] T 44d3c8472d1543938d1bbbfd85b79447 P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:58.463585 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 5351479a8993455baa868e09bdd7530e (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8384a6fd4134f569415b3c47c642ec9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:58.463831 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5351479a8993455baa868e09bdd7530e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:58.464478 24878 tablet_service.cc:1467] Processing CreateTablet for tablet 44d3c8472d1543938d1bbbfd85b79447 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8384a6fd4134f569415b3c47c642ec9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:58.464694 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d3c8472d1543938d1bbbfd85b79447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:58.464723 25684 tablet_bootstrap.cc:492] T 44d3c8472d1543938d1bbbfd85b79447 P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:08:58.464895 25684 ts_tablet_manager.cc:1367] T 44d3c8472d1543938d1bbbfd85b79447 P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:58.465332 25684 raft_consensus.cc:348] T 44d3c8472d1543938d1bbbfd85b79447 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:58.465435 25684 raft_consensus.cc:374] T 44d3c8472d1543938d1bbbfd85b79447 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:58.465445 25689 tablet_bootstrap.cc:492] T 94b711edc7694340a0c4a68ef178d8fe P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:08:58.465469 25684 raft_consensus.cc:729] T 44d3c8472d1543938d1bbbfd85b79447 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:08:58.465586 25684 consensus_queue.cc:260] T 44d3c8472d1543938d1bbbfd85b79447 P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:58.465873 25684 ts_tablet_manager.cc:1398] T 44d3c8472d1543938d1bbbfd85b79447 P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:58.466439 25143 tablet_service.cc:1467] Processing CreateTablet for tablet 5351479a8993455baa868e09bdd7530e (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8384a6fd4134f569415b3c47c642ec9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:58.466537 24481 tablet_service.cc:1467] Processing CreateTablet for tablet 94b711edc7694340a0c4a68ef178d8fe (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8384a6fd4134f569415b3c47c642ec9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:58.466668 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5351479a8993455baa868e09bdd7530e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:58.466818 24481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94b711edc7694340a0c4a68ef178d8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:58.467576 25690 tablet_bootstrap.cc:492] T 5351479a8993455baa868e09bdd7530e P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:08:58.468081 24613 tablet_service.cc:1467] Processing CreateTablet for tablet 5351479a8993455baa868e09bdd7530e (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8384a6fd4134f569415b3c47c642ec9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:58.468313 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5351479a8993455baa868e09bdd7530e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:58.468461 25690 tablet_bootstrap.cc:654] T 5351479a8993455baa868e09bdd7530e P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:58.469141 25689 tablet_bootstrap.cc:654] T 94b711edc7694340a0c4a68ef178d8fe P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:58.469316 25684 tablet_bootstrap.cc:492] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:08:58.470182 25687 tablet_bootstrap.cc:492] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:08:58.470417 25686 tablet_bootstrap.cc:492] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:08:58.470744 25684 tablet_bootstrap.cc:654] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:58.471194 25686 tablet_bootstrap.cc:654] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:58.471698 25687 tablet_bootstrap.cc:654] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:58.472038 25686 tablet_bootstrap.cc:492] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:08:58.472115 25686 ts_tablet_manager.cc:1367] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:58.472226 25689 tablet_bootstrap.cc:492] T 94b711edc7694340a0c4a68ef178d8fe P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:08:58.472393 25689 ts_tablet_manager.cc:1367] T 94b711edc7694340a0c4a68ef178d8fe P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:08:58.472520 25686 raft_consensus.cc:348] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:58.472631 25686 raft_consensus.cc:374] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:58.472663 25686 raft_consensus.cc:729] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:08:58.472790 25690 tablet_bootstrap.cc:492] T 5351479a8993455baa868e09bdd7530e P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:08:58.472896 25690 ts_tablet_manager.cc:1367] T 5351479a8993455baa868e09bdd7530e P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:08:58.472851 25689 raft_consensus.cc:348] T 94b711edc7694340a0c4a68ef178d8fe P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:58.472947 25689 raft_consensus.cc:374] T 94b711edc7694340a0c4a68ef178d8fe P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:58.472787 25686 consensus_queue.cc:260] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:58.473157 25689 raft_consensus.cc:729] T 94b711edc7694340a0c4a68ef178d8fe P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:08:58.473212 25686 ts_tablet_manager.cc:1398] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:58.473325 25689 consensus_queue.cc:260] T 94b711edc7694340a0c4a68ef178d8fe P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:58.473452 25686 tablet_bootstrap.cc:492] T 5351479a8993455baa868e09bdd7530e P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:08:58.473680 25689 ts_tablet_manager.cc:1398] T 94b711edc7694340a0c4a68ef178d8fe P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:58.474125 25686 tablet_bootstrap.cc:654] T 5351479a8993455baa868e09bdd7530e P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:58.474277 25684 tablet_bootstrap.cc:492] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:08:58.474363 25684 ts_tablet_manager.cc:1367] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:08:58.473599 25690 raft_consensus.cc:348] T 5351479a8993455baa868e09bdd7530e P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:58.474494 25690 raft_consensus.cc:374] T 5351479a8993455baa868e09bdd7530e P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:58.474534 25690 raft_consensus.cc:729] T 5351479a8993455baa868e09bdd7530e P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:08:58.474675 25690 consensus_queue.cc:260] T 5351479a8993455baa868e09bdd7530e P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:58.474769 25684 raft_consensus.cc:348] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:58.474850 25684 raft_consensus.cc:374] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:58.474879 25684 raft_consensus.cc:729] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:08:58.474983 25684 consensus_queue.cc:260] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:58.475198 25690 ts_tablet_manager.cc:1398] T 5351479a8993455baa868e09bdd7530e P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:58.475232 25684 ts_tablet_manager.cc:1398] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:58.475356 25690 tablet_bootstrap.cc:492] T 44d3c8472d1543938d1bbbfd85b79447 P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:08:58.476657 25687 tablet_bootstrap.cc:492] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:08:58.476872 25690 tablet_bootstrap.cc:654] T 44d3c8472d1543938d1bbbfd85b79447 P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:58.477538 25687 ts_tablet_manager.cc:1367] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:08:58.477702 25691 tablet_bootstrap.cc:492] T 94b711edc7694340a0c4a68ef178d8fe P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:08:58.477725 25686 tablet_bootstrap.cc:492] T 5351479a8993455baa868e09bdd7530e P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:08:58.477799 25686 ts_tablet_manager.cc:1367] T 5351479a8993455baa868e09bdd7530e P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:08:58.478210 25686 raft_consensus.cc:348] T 5351479a8993455baa868e09bdd7530e P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:58.478322 25686 raft_consensus.cc:374] T 5351479a8993455baa868e09bdd7530e P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:58.478358 25686 raft_consensus.cc:729] T 5351479a8993455baa868e09bdd7530e P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:08:58.478482 25686 consensus_queue.cc:260] T 5351479a8993455baa868e09bdd7530e P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:58.478471 25687 raft_consensus.cc:348] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:58.478667 25687 raft_consensus.cc:374] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:58.478742 25687 raft_consensus.cc:729] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:08:58.478766 25686 ts_tablet_manager.cc:1398] T 5351479a8993455baa868e09bdd7530e P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:58.478925 25691 tablet_bootstrap.cc:654] T 94b711edc7694340a0c4a68ef178d8fe P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:58.478967 25687 consensus_queue.cc:260] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:58.479279 25687 ts_tablet_manager.cc:1398] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:58.480000 25690 tablet_bootstrap.cc:492] T 44d3c8472d1543938d1bbbfd85b79447 P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:08:58.480111 25690 ts_tablet_manager.cc:1367] T 44d3c8472d1543938d1bbbfd85b79447 P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:08:58.480537 25691 tablet_bootstrap.cc:492] T 94b711edc7694340a0c4a68ef178d8fe P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:08:58.480535 25690 raft_consensus.cc:348] T 44d3c8472d1543938d1bbbfd85b79447 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:58.480623 25691 ts_tablet_manager.cc:1367] T 94b711edc7694340a0c4a68ef178d8fe P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:58.480628 25690 raft_consensus.cc:374] T 44d3c8472d1543938d1bbbfd85b79447 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:58.480655 25690 raft_consensus.cc:729] T 44d3c8472d1543938d1bbbfd85b79447 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:08:58.480789 25690 consensus_queue.cc:260] T 44d3c8472d1543938d1bbbfd85b79447 P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:58.481069 25691 raft_consensus.cc:348] T 94b711edc7694340a0c4a68ef178d8fe P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:58.481138 25690 ts_tablet_manager.cc:1398] T 44d3c8472d1543938d1bbbfd85b79447 P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:58.481169 25691 raft_consensus.cc:374] T 94b711edc7694340a0c4a68ef178d8fe P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:58.481197 25691 raft_consensus.cc:729] T 94b711edc7694340a0c4a68ef178d8fe P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:08:58.481319 25691 consensus_queue.cc:260] T 94b711edc7694340a0c4a68ef178d8fe P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:58.481585 25691 ts_tablet_manager.cc:1398] T 94b711edc7694340a0c4a68ef178d8fe P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:58.580900 25589 raft_consensus.cc:482] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:58.581039 25589 raft_consensus.cc:504] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:58.581459 25589 leader_election.cc:288] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:08:58.581854 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b711edc7694340a0c4a68ef178d8fe" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:08:58.581858 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b711edc7694340a0c4a68ef178d8fe" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" is_pre_election: true
I20240501 14:08:58.581991 24765 raft_consensus.cc:2454] T 94b711edc7694340a0c4a68ef178d8fe P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:08:58.582001 24502 raft_consensus.cc:2454] T 94b711edc7694340a0c4a68ef178d8fe P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:08:58.582276 24963 leader_election.cc:302] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:08:58.582475 25589 raft_consensus.cc:2781] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:58.582544 25589 raft_consensus.cc:482] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:58.582584 25589 raft_consensus.cc:3037] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.611918 25674 raft_consensus.cc:482] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:58.612053 25674 raft_consensus.cc:504] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:58.612512 25674 leader_election.cc:288] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:08:58.612535 25589 raft_consensus.cc:504] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:58.612968 25589 leader_election.cc:288] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Requested vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:08:58.612963 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5351479a8993455baa868e09bdd7530e" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6" is_pre_election: true
I20240501 14:08:58.613108 24897 raft_consensus.cc:2454] T 5351479a8993455baa868e09bdd7530e P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:08:58.613306 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5351479a8993455baa868e09bdd7530e" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3" is_pre_election: true
I20240501 14:08:58.613447 24634 raft_consensus.cc:2454] T 5351479a8993455baa868e09bdd7530e P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:08:58.613454 25098 leader_election.cc:302] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, c2691129fd82431bb867cb1307e009b6; no voters:
I20240501 14:08:58.613474 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b711edc7694340a0c4a68ef178d8fe" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:08:58.613588 24502 raft_consensus.cc:3037] T 94b711edc7694340a0c4a68ef178d8fe P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.613620 25674 raft_consensus.cc:2781] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:58.613662 25674 raft_consensus.cc:482] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:58.613693 25674 raft_consensus.cc:3037] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.614298 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b711edc7694340a0c4a68ef178d8fe" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
I20240501 14:08:58.614487 24765 raft_consensus.cc:3037] T 94b711edc7694340a0c4a68ef178d8fe P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.614629 24502 raft_consensus.cc:2454] T 94b711edc7694340a0c4a68ef178d8fe P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:08:58.614923 25674 raft_consensus.cc:504] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:58.615005 24964 leader_election.cc:302] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e375044d63649c097759c4b667eff79, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:08:58.615171 25589 raft_consensus.cc:2781] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:58.615260 25589 raft_consensus.cc:686] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [term 1 LEADER]: Becoming Leader. State: Replica: 7e375044d63649c097759c4b667eff79, State: Running, Role: LEADER
I20240501 14:08:58.615267 25674 leader_election.cc:288] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Requested vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:08:58.615394 25589 consensus_queue.cc:237] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:58.615448 24765 raft_consensus.cc:2454] T 94b711edc7694340a0c4a68ef178d8fe P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:08:58.616016 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5351479a8993455baa868e09bdd7530e" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6"
I20240501 14:08:58.616073 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5351479a8993455baa868e09bdd7530e" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3"
I20240501 14:08:58.616140 24897 raft_consensus.cc:3037] T 5351479a8993455baa868e09bdd7530e P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.616190 24634 raft_consensus.cc:3037] T 5351479a8993455baa868e09bdd7530e P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.617489 24360 catalog_manager.cc:5484] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e375044d63649c097759c4b667eff79 (127.23.191.69). New cstate: current_term: 1 leader_uuid: "7e375044d63649c097759c4b667eff79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:58.706732 25669 raft_consensus.cc:482] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:58.706853 25669 raft_consensus.cc:504] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:58.707226 25669 leader_election.cc:288] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343), c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971)
I20240501 14:08:58.707572 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d3c8472d1543938d1bbbfd85b79447" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865" is_pre_election: true
I20240501 14:08:58.707720 25163 raft_consensus.cc:2454] T 44d3c8472d1543938d1bbbfd85b79447 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 0.
I20240501 14:08:58.707686 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d3c8472d1543938d1bbbfd85b79447" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6" is_pre_election: true
I20240501 14:08:58.707808 24898 raft_consensus.cc:2454] T 44d3c8472d1543938d1bbbfd85b79447 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 0.
I20240501 14:08:58.708029 24569 leader_election.cc:302] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:58.708222 25669 raft_consensus.cc:2781] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:58.708313 25669 raft_consensus.cc:482] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:58.708360 25669 raft_consensus.cc:3037] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.711825 24897 raft_consensus.cc:2454] T 5351479a8993455baa868e09bdd7530e P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:08:58.711834 24634 raft_consensus.cc:2454] T 5351479a8993455baa868e09bdd7530e P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:08:58.712160 25095 leader_election.cc:302] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:58.712404 25674 raft_consensus.cc:2781] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:58.712510 25674 raft_consensus.cc:686] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [term 1 LEADER]: Becoming Leader. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Running, Role: LEADER
I20240501 14:08:58.712654 25674 consensus_queue.cc:237] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:58.712791 25669 raft_consensus.cc:504] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:58.713181 25669 leader_election.cc:288] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 election: Requested vote from peers 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343), c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971)
I20240501 14:08:58.713522 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d3c8472d1543938d1bbbfd85b79447" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6"
I20240501 14:08:58.713642 24897 raft_consensus.cc:3037] T 44d3c8472d1543938d1bbbfd85b79447 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.713912 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d3c8472d1543938d1bbbfd85b79447" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865"
I20240501 14:08:58.714010 25163 raft_consensus.cc:3037] T 44d3c8472d1543938d1bbbfd85b79447 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.714563 24897 raft_consensus.cc:2454] T 44d3c8472d1543938d1bbbfd85b79447 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 1.
I20240501 14:08:58.714854 25163 raft_consensus.cc:2454] T 44d3c8472d1543938d1bbbfd85b79447 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 1.
I20240501 14:08:58.714864 24570 leader_election.cc:302] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2691129fd82431bb867cb1307e009b6, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:58.715023 25669 raft_consensus.cc:2781] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:58.715327 25669 raft_consensus.cc:686] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 LEADER]: Becoming Leader. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Running, Role: LEADER
I20240501 14:08:58.715241 24360 catalog_manager.cc:5484] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70). New cstate: current_term: 1 leader_uuid: "786f81b7c57f452e9b463f721fc3e865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:58.715472 25669 consensus_queue.cc:237] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:08:58.717193 24357 catalog_manager.cc:5484] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66). New cstate: current_term: 1 leader_uuid: "f68136de04c94e5cb4a5f99881f02ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: UNKNOWN } } }
W20240501 14:08:58.867353 25703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:58.867579 25703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:58.870556 25703 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 14:08:58.971236 24765 raft_consensus.cc:1261] T 94b711edc7694340a0c4a68ef178d8fe P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Refusing update from remote peer 7e375044d63649c097759c4b667eff79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:58.971691 24502 raft_consensus.cc:1261] T 94b711edc7694340a0c4a68ef178d8fe P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Refusing update from remote peer 7e375044d63649c097759c4b667eff79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:58.972263 25163 raft_consensus.cc:1261] T 44d3c8472d1543938d1bbbfd85b79447 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Refusing update from remote peer f68136de04c94e5cb4a5f99881f02ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:58.972376 24897 raft_consensus.cc:1261] T 44d3c8472d1543938d1bbbfd85b79447 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Refusing update from remote peer f68136de04c94e5cb4a5f99881f02ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:58.972620 25589 consensus_queue.cc:1035] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:58.972666 25669 consensus_queue.cc:1035] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:58.973659 25589 consensus_queue.cc:1035] T 94b711edc7694340a0c4a68ef178d8fe P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:58.974334 25669 consensus_queue.cc:1035] T 44d3c8472d1543938d1bbbfd85b79447 P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:58.983330 24634 raft_consensus.cc:1261] T 5351479a8993455baa868e09bdd7530e P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:58.983991 25263 consensus_queue.cc:1035] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:58.991784 24897 raft_consensus.cc:1261] T 5351479a8993455baa868e09bdd7530e P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:58.992332 25263 consensus_queue.cc:1035] T 5351479a8993455baa868e09bdd7530e P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.254 ms
time per row: 0.0621271 ms
I20240501 14:08:59.041082 24356 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43068:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:59.046782 24746 tablet_service.cc:1467] Processing CreateTablet for tablet b8afe2fe38f84644a14abac37af3fed0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=b826ceb0ae6446a8a3444dd7576cc5c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.047116 24746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8afe2fe38f84644a14abac37af3fed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.047300 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef5de6cff3c4bc8856563f25222605f (DEFAULT_TABLE table=rebalance_test_table_6 [id=b826ceb0ae6446a8a3444dd7576cc5c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.047572 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef5de6cff3c4bc8856563f25222605f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.047683 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 84643234d6424fab8eef66f4ae2eb254 (DEFAULT_TABLE table=rebalance_test_table_6 [id=b826ceb0ae6446a8a3444dd7576cc5c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.047695 25009 tablet_service.cc:1467] Processing CreateTablet for tablet b8afe2fe38f84644a14abac37af3fed0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=b826ceb0ae6446a8a3444dd7576cc5c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.047896 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84643234d6424fab8eef66f4ae2eb254. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.047945 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8afe2fe38f84644a14abac37af3fed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.049585 24613 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef5de6cff3c4bc8856563f25222605f (DEFAULT_TABLE table=rebalance_test_table_6 [id=b826ceb0ae6446a8a3444dd7576cc5c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.049875 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef5de6cff3c4bc8856563f25222605f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.050153 25763 tablet_bootstrap.cc:492] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:08:59.050981 25763 tablet_bootstrap.cc:654] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.051990 25763 tablet_bootstrap.cc:492] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:08:59.052003 25766 tablet_bootstrap.cc:492] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:08:59.052063 25763 ts_tablet_manager.cc:1367] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:59.052224 25143 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef5de6cff3c4bc8856563f25222605f (DEFAULT_TABLE table=rebalance_test_table_6 [id=b826ceb0ae6446a8a3444dd7576cc5c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.052471 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef5de6cff3c4bc8856563f25222605f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.052501 25763 raft_consensus.cc:348] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:59.052592 25763 raft_consensus.cc:374] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.052659 25763 raft_consensus.cc:729] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.052760 25766 tablet_bootstrap.cc:654] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.052888 25763 consensus_queue.cc:260] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:59.053894 25763 ts_tablet_manager.cc:1398] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:59.054037 25763 tablet_bootstrap.cc:492] T 84643234d6424fab8eef66f4ae2eb254 P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:08:59.054915 25010 tablet_service.cc:1467] Processing CreateTablet for tablet 84643234d6424fab8eef66f4ae2eb254 (DEFAULT_TABLE table=rebalance_test_table_6 [id=b826ceb0ae6446a8a3444dd7576cc5c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.055153 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84643234d6424fab8eef66f4ae2eb254. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.055317 25763 tablet_bootstrap.cc:654] T 84643234d6424fab8eef66f4ae2eb254 P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.055898 25768 tablet_bootstrap.cc:492] T 8ef5de6cff3c4bc8856563f25222605f P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:08:59.056016 25573 raft_consensus.cc:482] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:59.056089 25573 raft_consensus.cc:504] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:59.056448 25573 leader_election.cc:288] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:08:59.056568 25763 tablet_bootstrap.cc:492] T 84643234d6424fab8eef66f4ae2eb254 P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:08:59.056636 25763 ts_tablet_manager.cc:1367] T 84643234d6424fab8eef66f4ae2eb254 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:08:59.056874 25768 tablet_bootstrap.cc:654] T 8ef5de6cff3c4bc8856563f25222605f P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.057082 25763 raft_consensus.cc:348] T 84643234d6424fab8eef66f4ae2eb254 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.057168 25763 raft_consensus.cc:374] T 84643234d6424fab8eef66f4ae2eb254 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.057197 25763 raft_consensus.cc:729] T 84643234d6424fab8eef66f4ae2eb254 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.057335 25770 tablet_bootstrap.cc:492] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:08:59.057328 25763 consensus_queue.cc:260] T 84643234d6424fab8eef66f4ae2eb254 P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.057654 25763 ts_tablet_manager.cc:1398] T 84643234d6424fab8eef66f4ae2eb254 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:59.057677 24481 tablet_service.cc:1467] Processing CreateTablet for tablet b8afe2fe38f84644a14abac37af3fed0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=b826ceb0ae6446a8a3444dd7576cc5c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
W20240501 14:08:59.057865 24699 leader_election.cc:341] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635): Not found: Tablet not found: b8afe2fe38f84644a14abac37af3fed0
I20240501 14:08:59.057950 24481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8afe2fe38f84644a14abac37af3fed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.058804 25770 tablet_bootstrap.cc:654] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.059854 25764 tablet_bootstrap.cc:492] T 8ef5de6cff3c4bc8856563f25222605f P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:08:59.060268 25770 tablet_bootstrap.cc:492] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:08:59.060406 25770 ts_tablet_manager.cc:1367] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:59.060607 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8afe2fe38f84644a14abac37af3fed0" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:08:59.060928 25770 raft_consensus.cc:348] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.061086 25770 raft_consensus.cc:374] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.057680 24482 tablet_service.cc:1467] Processing CreateTablet for tablet 84643234d6424fab8eef66f4ae2eb254 (DEFAULT_TABLE table=rebalance_test_table_6 [id=b826ceb0ae6446a8a3444dd7576cc5c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.061158 25770 raft_consensus.cc:729] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.061300 24482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84643234d6424fab8eef66f4ae2eb254. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.061326 25770 consensus_queue.cc:260] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.061367 25764 tablet_bootstrap.cc:654] T 8ef5de6cff3c4bc8856563f25222605f P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.061690 25770 ts_tablet_manager.cc:1398] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:59.061812 25766 tablet_bootstrap.cc:492] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:08:59.061828 25770 tablet_bootstrap.cc:492] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:08:59.062151 25766 ts_tablet_manager.cc:1367] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240501 14:08:59.062577 25770 tablet_bootstrap.cc:654] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.062631 25766 raft_consensus.cc:348] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.063390 25766 raft_consensus.cc:374] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.063565 25770 tablet_bootstrap.cc:492] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:08:59.063637 25770 ts_tablet_manager.cc:1367] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:59.063453 25766 raft_consensus.cc:729] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
W20240501 14:08:59.063813 24700 leader_election.cc:341] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:08:59.063822 25766 consensus_queue.cc:260] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.063880 24700 leader_election.cc:302] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7; no voters: 7e375044d63649c097759c4b667eff79, fd8ec7cff3a147e5adfc4eb329a23895
I20240501 14:08:59.064020 25770 raft_consensus.cc:348] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:59.064101 25770 raft_consensus.cc:374] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.064129 25770 raft_consensus.cc:729] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.064182 25766 ts_tablet_manager.cc:1398] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:08:59.064229 25770 consensus_queue.cc:260] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:59.064339 25573 raft_consensus.cc:2726] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:08:59.064473 25770 ts_tablet_manager.cc:1398] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:59.065187 25774 tablet_bootstrap.cc:492] T b8afe2fe38f84644a14abac37af3fed0 P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:08:59.065691 25768 tablet_bootstrap.cc:492] T 8ef5de6cff3c4bc8856563f25222605f P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:08:59.065774 25768 ts_tablet_manager.cc:1367] T 8ef5de6cff3c4bc8856563f25222605f P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20240501 14:08:59.066429 25768 raft_consensus.cc:348] T 8ef5de6cff3c4bc8856563f25222605f P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.066520 25768 raft_consensus.cc:374] T 8ef5de6cff3c4bc8856563f25222605f P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.066550 25768 raft_consensus.cc:729] T 8ef5de6cff3c4bc8856563f25222605f P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.066558 25774 tablet_bootstrap.cc:654] T b8afe2fe38f84644a14abac37af3fed0 P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.066671 25768 consensus_queue.cc:260] T 8ef5de6cff3c4bc8856563f25222605f P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.066932 25768 ts_tablet_manager.cc:1398] T 8ef5de6cff3c4bc8856563f25222605f P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:59.067739 25764 tablet_bootstrap.cc:492] T 8ef5de6cff3c4bc8856563f25222605f P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:08:59.067827 25764 ts_tablet_manager.cc:1367] T 8ef5de6cff3c4bc8856563f25222605f P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:08:59.068218 25764 raft_consensus.cc:348] T 8ef5de6cff3c4bc8856563f25222605f P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.068322 25764 raft_consensus.cc:374] T 8ef5de6cff3c4bc8856563f25222605f P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.068344 25764 raft_consensus.cc:729] T 8ef5de6cff3c4bc8856563f25222605f P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.068465 25764 consensus_queue.cc:260] T 8ef5de6cff3c4bc8856563f25222605f P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.068706 25764 ts_tablet_manager.cc:1398] T 8ef5de6cff3c4bc8856563f25222605f P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:59.069413 25774 tablet_bootstrap.cc:492] T b8afe2fe38f84644a14abac37af3fed0 P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:08:59.069495 25774 ts_tablet_manager.cc:1367] T b8afe2fe38f84644a14abac37af3fed0 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:59.069857 25774 raft_consensus.cc:348] T b8afe2fe38f84644a14abac37af3fed0 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:59.069931 25774 raft_consensus.cc:374] T b8afe2fe38f84644a14abac37af3fed0 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.069948 25774 raft_consensus.cc:729] T b8afe2fe38f84644a14abac37af3fed0 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.070065 25774 consensus_queue.cc:260] T b8afe2fe38f84644a14abac37af3fed0 P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:59.070334 25774 ts_tablet_manager.cc:1398] T b8afe2fe38f84644a14abac37af3fed0 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:59.070485 25774 tablet_bootstrap.cc:492] T 84643234d6424fab8eef66f4ae2eb254 P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:08:59.071185 25774 tablet_bootstrap.cc:654] T 84643234d6424fab8eef66f4ae2eb254 P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.072494 25774 tablet_bootstrap.cc:492] T 84643234d6424fab8eef66f4ae2eb254 P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:08:59.072561 25774 ts_tablet_manager.cc:1367] T 84643234d6424fab8eef66f4ae2eb254 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:59.072975 25774 raft_consensus.cc:348] T 84643234d6424fab8eef66f4ae2eb254 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.073073 25774 raft_consensus.cc:374] T 84643234d6424fab8eef66f4ae2eb254 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.073103 25774 raft_consensus.cc:729] T 84643234d6424fab8eef66f4ae2eb254 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.073212 25774 consensus_queue.cc:260] T 84643234d6424fab8eef66f4ae2eb254 P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.073446 25774 ts_tablet_manager.cc:1398] T 84643234d6424fab8eef66f4ae2eb254 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:59.102869 25752 raft_consensus.cc:482] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:59.103024 25752 raft_consensus.cc:504] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:59.103457 25752 leader_election.cc:288] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:08:59.103865 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8afe2fe38f84644a14abac37af3fed0" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" is_pre_election: true
I20240501 14:08:59.103871 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8afe2fe38f84644a14abac37af3fed0" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:08:59.104020 24765 raft_consensus.cc:2454] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:08:59.104017 24501 raft_consensus.cc:2454] T b8afe2fe38f84644a14abac37af3fed0 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:08:59.104310 24964 leader_election.cc:302] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e375044d63649c097759c4b667eff79, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:08:59.104501 25752 raft_consensus.cc:2781] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:59.104568 25752 raft_consensus.cc:482] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:59.104606 25752 raft_consensus.cc:3037] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.109189 25752 raft_consensus.cc:504] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:59.109586 25752 leader_election.cc:288] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Requested vote from peers 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:08:59.109896 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8afe2fe38f84644a14abac37af3fed0" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
I20240501 14:08:59.110019 24765 raft_consensus.cc:3037] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.110009 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8afe2fe38f84644a14abac37af3fed0" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:08:59.110121 24501 raft_consensus.cc:3037] T b8afe2fe38f84644a14abac37af3fed0 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.110962 24501 raft_consensus.cc:2454] T b8afe2fe38f84644a14abac37af3fed0 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:08:59.110962 24765 raft_consensus.cc:2454] T b8afe2fe38f84644a14abac37af3fed0 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:08:59.111265 24963 leader_election.cc:302] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:08:59.111438 25752 raft_consensus.cc:2781] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:59.111517 25752 raft_consensus.cc:686] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [term 1 LEADER]: Becoming Leader. State: Replica: 7e375044d63649c097759c4b667eff79, State: Running, Role: LEADER
I20240501 14:08:59.111653 25752 consensus_queue.cc:237] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:08:59.113405 24358 catalog_manager.cc:5484] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e375044d63649c097759c4b667eff79 (127.23.191.69). New cstate: current_term: 1 leader_uuid: "7e375044d63649c097759c4b667eff79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:59.150817 25752 raft_consensus.cc:482] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:59.150938 25752 raft_consensus.cc:504] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.151327 25752 leader_election.cc:288] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:08:59.151693 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84643234d6424fab8eef66f4ae2eb254" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:08:59.151741 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84643234d6424fab8eef66f4ae2eb254" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" is_pre_election: true
I20240501 14:08:59.151852 24501 raft_consensus.cc:2454] T 84643234d6424fab8eef66f4ae2eb254 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:08:59.151906 24765 raft_consensus.cc:2454] T 84643234d6424fab8eef66f4ae2eb254 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:08:59.152101 24964 leader_election.cc:302] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e375044d63649c097759c4b667eff79, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:08:59.152262 25752 raft_consensus.cc:2781] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:59.152328 25752 raft_consensus.cc:482] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:59.152366 25752 raft_consensus.cc:3037] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.206354 25752 raft_consensus.cc:504] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.206892 25752 leader_election.cc:288] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Requested vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:08:59.207352 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84643234d6424fab8eef66f4ae2eb254" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:08:59.207351 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84643234d6424fab8eef66f4ae2eb254" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
I20240501 14:08:59.207504 24501 raft_consensus.cc:3037] T 84643234d6424fab8eef66f4ae2eb254 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.207504 24765 raft_consensus.cc:3037] T 84643234d6424fab8eef66f4ae2eb254 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.293913 25669 raft_consensus.cc:482] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:59.294037 25669 raft_consensus.cc:504] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.294390 25669 leader_election.cc:288] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343)
I20240501 14:08:59.294756 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef5de6cff3c4bc8856563f25222605f" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6" is_pre_election: true
I20240501 14:08:59.294813 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef5de6cff3c4bc8856563f25222605f" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865" is_pre_election: true
I20240501 14:08:59.294894 24897 raft_consensus.cc:2454] T 8ef5de6cff3c4bc8856563f25222605f P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 0.
I20240501 14:08:59.294965 25163 raft_consensus.cc:2454] T 8ef5de6cff3c4bc8856563f25222605f P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 0.
I20240501 14:08:59.295179 24570 leader_election.cc:302] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2691129fd82431bb867cb1307e009b6, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:59.295358 25669 raft_consensus.cc:2781] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:59.295418 25669 raft_consensus.cc:482] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:59.295449 25669 raft_consensus.cc:3037] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.306007 24501 raft_consensus.cc:2454] T 84643234d6424fab8eef66f4ae2eb254 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:08:59.306005 24765 raft_consensus.cc:2454] T 84643234d6424fab8eef66f4ae2eb254 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:08:59.306007 25669 raft_consensus.cc:504] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.306391 25669 leader_election.cc:288] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 election: Requested vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343)
I20240501 14:08:59.306478 24963 leader_election.cc:302] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:08:59.306656 25752 raft_consensus.cc:2781] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:59.306756 25752 raft_consensus.cc:686] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [term 1 LEADER]: Becoming Leader. State: Replica: 7e375044d63649c097759c4b667eff79, State: Running, Role: LEADER
I20240501 14:08:59.306774 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef5de6cff3c4bc8856563f25222605f" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6"
I20240501 14:08:59.306882 24897 raft_consensus.cc:3037] T 8ef5de6cff3c4bc8856563f25222605f P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.306888 25752 consensus_queue.cc:237] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.307014 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef5de6cff3c4bc8856563f25222605f" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865"
I20240501 14:08:59.307129 25163 raft_consensus.cc:3037] T 8ef5de6cff3c4bc8856563f25222605f P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.307868 24897 raft_consensus.cc:2454] T 8ef5de6cff3c4bc8856563f25222605f P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 1.
I20240501 14:08:59.308001 25163 raft_consensus.cc:2454] T 8ef5de6cff3c4bc8856563f25222605f P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 1.
I20240501 14:08:59.308202 24570 leader_election.cc:302] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2691129fd82431bb867cb1307e009b6, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:59.308385 25669 raft_consensus.cc:2781] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:59.308470 25669 raft_consensus.cc:686] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [term 1 LEADER]: Becoming Leader. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Running, Role: LEADER
I20240501 14:08:59.308596 25669 consensus_queue.cc:237] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.309489 24358 catalog_manager.cc:5484] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e375044d63649c097759c4b667eff79 (127.23.191.69). New cstate: current_term: 1 leader_uuid: "7e375044d63649c097759c4b667eff79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:59.310294 24356 catalog_manager.cc:5484] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66). New cstate: current_term: 1 leader_uuid: "f68136de04c94e5cb4a5f99881f02ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: HEALTHY } } }
W20240501 14:08:59.462492 25783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:59.462749 25783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:59.465947 25783 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 14:08:59.559265 25589 consensus_queue.cc:1035] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:59.565096 25752 consensus_queue.cc:1035] T b8afe2fe38f84644a14abac37af3fed0 P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:59.583915 24765 raft_consensus.cc:1261] T 84643234d6424fab8eef66f4ae2eb254 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Refusing update from remote peer 7e375044d63649c097759c4b667eff79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:59.584995 24501 raft_consensus.cc:1261] T 84643234d6424fab8eef66f4ae2eb254 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Refusing update from remote peer 7e375044d63649c097759c4b667eff79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:59.585249 25817 consensus_queue.cc:1035] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:59.588083 25817 consensus_queue.cc:1035] T 84643234d6424fab8eef66f4ae2eb254 P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:59.592468 24897 raft_consensus.cc:1261] T 8ef5de6cff3c4bc8856563f25222605f P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Refusing update from remote peer f68136de04c94e5cb4a5f99881f02ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:59.592746 25163 raft_consensus.cc:1261] T 8ef5de6cff3c4bc8856563f25222605f P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Refusing update from remote peer f68136de04c94e5cb4a5f99881f02ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:59.592969 25583 consensus_queue.cc:1035] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:59.593183 25669 consensus_queue.cc:1035] T 8ef5de6cff3c4bc8856563f25222605f P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 139.239 ms
time per row: 0.0696196 ms
I20240501 14:08:59.648394 24359 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43068:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:59.654045 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 78a1abcad3534f668299a245393d9022 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2cd07df468c94f81b45cd19eeafc9ad5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.654350 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a1abcad3534f668299a245393d9022. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.654451 24878 tablet_service.cc:1467] Processing CreateTablet for tablet 071272f7e339410890c23fb7e77465c1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2cd07df468c94f81b45cd19eeafc9ad5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.654742 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071272f7e339410890c23fb7e77465c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.655006 24613 tablet_service.cc:1467] Processing CreateTablet for tablet 78a1abcad3534f668299a245393d9022 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2cd07df468c94f81b45cd19eeafc9ad5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.655159 25143 tablet_service.cc:1467] Processing CreateTablet for tablet 78a1abcad3534f668299a245393d9022 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2cd07df468c94f81b45cd19eeafc9ad5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.655263 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a1abcad3534f668299a245393d9022. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.655428 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a1abcad3534f668299a245393d9022. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.657073 25838 tablet_bootstrap.cc:492] T 78a1abcad3534f668299a245393d9022 P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:08:59.657485 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 7b311d8b2b3f40f88c1580e9c24fcd4b (DEFAULT_TABLE table=rebalance_test_table_7 [id=2cd07df468c94f81b45cd19eeafc9ad5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.657593 25840 tablet_bootstrap.cc:492] T 78a1abcad3534f668299a245393d9022 P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:08:59.657743 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b311d8b2b3f40f88c1580e9c24fcd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.657775 25009 tablet_service.cc:1467] Processing CreateTablet for tablet 7b311d8b2b3f40f88c1580e9c24fcd4b (DEFAULT_TABLE table=rebalance_test_table_7 [id=2cd07df468c94f81b45cd19eeafc9ad5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.658025 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b311d8b2b3f40f88c1580e9c24fcd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.658272 25840 tablet_bootstrap.cc:654] T 78a1abcad3534f668299a245393d9022 P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.659092 25840 tablet_bootstrap.cc:492] T 78a1abcad3534f668299a245393d9022 P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:08:59.659150 25840 ts_tablet_manager.cc:1367] T 78a1abcad3534f668299a245393d9022 P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:59.659485 25840 raft_consensus.cc:348] T 78a1abcad3534f668299a245393d9022 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:59.659559 25840 raft_consensus.cc:374] T 78a1abcad3534f668299a245393d9022 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.659577 25840 raft_consensus.cc:729] T 78a1abcad3534f668299a245393d9022 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.659682 25840 consensus_queue.cc:260] T 78a1abcad3534f668299a245393d9022 P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:59.660128 25840 ts_tablet_manager.cc:1398] T 78a1abcad3534f668299a245393d9022 P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:59.660567 24482 tablet_service.cc:1467] Processing CreateTablet for tablet 7b311d8b2b3f40f88c1580e9c24fcd4b (DEFAULT_TABLE table=rebalance_test_table_7 [id=2cd07df468c94f81b45cd19eeafc9ad5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.660666 24613 tablet_service.cc:1467] Processing CreateTablet for tablet 071272f7e339410890c23fb7e77465c1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2cd07df468c94f81b45cd19eeafc9ad5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.660878 24482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b311d8b2b3f40f88c1580e9c24fcd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.660919 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071272f7e339410890c23fb7e77465c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.660977 25143 tablet_service.cc:1467] Processing CreateTablet for tablet 071272f7e339410890c23fb7e77465c1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2cd07df468c94f81b45cd19eeafc9ad5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:59.661190 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071272f7e339410890c23fb7e77465c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.661562 25838 tablet_bootstrap.cc:654] T 78a1abcad3534f668299a245393d9022 P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.662387 25842 tablet_bootstrap.cc:492] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:08:59.663033 25845 tablet_bootstrap.cc:492] T 7b311d8b2b3f40f88c1580e9c24fcd4b P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:08:59.663408 25842 tablet_bootstrap.cc:654] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.664657 25843 tablet_bootstrap.cc:492] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:08:59.664486 25845 tablet_bootstrap.cc:654] T 7b311d8b2b3f40f88c1580e9c24fcd4b P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.664947 25838 tablet_bootstrap.cc:492] T 78a1abcad3534f668299a245393d9022 P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:08:59.665025 25838 ts_tablet_manager.cc:1367] T 78a1abcad3534f668299a245393d9022 P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:08:59.665462 25840 tablet_bootstrap.cc:492] T 071272f7e339410890c23fb7e77465c1 P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:08:59.665688 25838 raft_consensus.cc:348] T 78a1abcad3534f668299a245393d9022 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:59.665840 25843 tablet_bootstrap.cc:654] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.665863 25838 raft_consensus.cc:374] T 78a1abcad3534f668299a245393d9022 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.665917 25838 raft_consensus.cc:729] T 78a1abcad3534f668299a245393d9022 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.666091 25838 consensus_queue.cc:260] T 78a1abcad3534f668299a245393d9022 P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:59.666428 25838 ts_tablet_manager.cc:1398] T 78a1abcad3534f668299a245393d9022 P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:59.666608 25839 tablet_bootstrap.cc:492] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:08:59.666642 25838 tablet_bootstrap.cc:492] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:08:59.666900 25840 tablet_bootstrap.cc:654] T 071272f7e339410890c23fb7e77465c1 P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.667045 25843 tablet_bootstrap.cc:492] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:08:59.667140 25843 ts_tablet_manager.cc:1367] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:59.667590 25838 tablet_bootstrap.cc:654] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.667590 25843 raft_consensus.cc:348] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.667697 25843 raft_consensus.cc:374] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.667730 25843 raft_consensus.cc:729] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.667847 25843 consensus_queue.cc:260] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.668148 25843 ts_tablet_manager.cc:1398] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:59.668946 25840 tablet_bootstrap.cc:492] T 071272f7e339410890c23fb7e77465c1 P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:08:59.669039 25840 ts_tablet_manager.cc:1367] T 071272f7e339410890c23fb7e77465c1 P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:59.669414 25838 tablet_bootstrap.cc:492] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:08:59.669425 25840 raft_consensus.cc:348] T 071272f7e339410890c23fb7e77465c1 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.669550 25838 ts_tablet_manager.cc:1367] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:59.669665 25840 raft_consensus.cc:374] T 071272f7e339410890c23fb7e77465c1 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.669715 25840 raft_consensus.cc:729] T 071272f7e339410890c23fb7e77465c1 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.669855 25840 consensus_queue.cc:260] T 071272f7e339410890c23fb7e77465c1 P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.670037 25838 raft_consensus.cc:348] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.670168 25840 ts_tablet_manager.cc:1398] T 071272f7e339410890c23fb7e77465c1 P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:59.670192 25838 raft_consensus.cc:374] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.670260 25842 tablet_bootstrap.cc:492] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:08:59.670310 25838 raft_consensus.cc:729] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.670392 25842 ts_tablet_manager.cc:1367] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:08:59.670538 25838 consensus_queue.cc:260] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.670856 25838 ts_tablet_manager.cc:1398] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:59.671612 25845 tablet_bootstrap.cc:492] T 7b311d8b2b3f40f88c1580e9c24fcd4b P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:08:59.671718 25845 ts_tablet_manager.cc:1367] T 7b311d8b2b3f40f88c1580e9c24fcd4b P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240501 14:08:59.672133 25845 raft_consensus.cc:348] T 7b311d8b2b3f40f88c1580e9c24fcd4b P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.672183 25842 raft_consensus.cc:348] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.672256 25845 raft_consensus.cc:374] T 7b311d8b2b3f40f88c1580e9c24fcd4b P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.672274 25842 raft_consensus.cc:374] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.672291 25845 raft_consensus.cc:729] T 7b311d8b2b3f40f88c1580e9c24fcd4b P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.672310 25842 raft_consensus.cc:729] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.672430 25842 consensus_queue.cc:260] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.672428 25845 consensus_queue.cc:260] T 7b311d8b2b3f40f88c1580e9c24fcd4b P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.672696 25845 ts_tablet_manager.cc:1398] T 7b311d8b2b3f40f88c1580e9c24fcd4b P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:59.680955 25842 ts_tablet_manager.cc:1398] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:08:59.682502 25839 tablet_bootstrap.cc:654] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.683879 25839 tablet_bootstrap.cc:492] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:08:59.683979 25839 ts_tablet_manager.cc:1367] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:08:59.684463 25839 raft_consensus.cc:348] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:59.684582 25839 raft_consensus.cc:374] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.684609 25839 raft_consensus.cc:729] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.684722 25839 consensus_queue.cc:260] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:59.685039 25839 ts_tablet_manager.cc:1398] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:59.685195 25839 tablet_bootstrap.cc:492] T 071272f7e339410890c23fb7e77465c1 P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:08:59.686053 25839 tablet_bootstrap.cc:654] T 071272f7e339410890c23fb7e77465c1 P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.687119 25839 tablet_bootstrap.cc:492] T 071272f7e339410890c23fb7e77465c1 P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:08:59.687196 25839 ts_tablet_manager.cc:1367] T 071272f7e339410890c23fb7e77465c1 P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:59.687592 25839 raft_consensus.cc:348] T 071272f7e339410890c23fb7e77465c1 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.687698 25839 raft_consensus.cc:374] T 071272f7e339410890c23fb7e77465c1 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.687758 25839 raft_consensus.cc:729] T 071272f7e339410890c23fb7e77465c1 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.687924 25839 consensus_queue.cc:260] T 071272f7e339410890c23fb7e77465c1 P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:08:59.688367 25839 ts_tablet_manager.cc:1398] T 071272f7e339410890c23fb7e77465c1 P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:59.720602 25589 raft_consensus.cc:482] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:59.720768 25589 raft_consensus.cc:504] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.721292 25589 leader_election.cc:288] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:08:59.721725 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b311d8b2b3f40f88c1580e9c24fcd4b" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:08:59.721859 24501 raft_consensus.cc:2454] T 7b311d8b2b3f40f88c1580e9c24fcd4b P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:08:59.721793 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b311d8b2b3f40f88c1580e9c24fcd4b" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" is_pre_election: true
I20240501 14:08:59.721975 24765 raft_consensus.cc:2454] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:08:59.722143 24964 leader_election.cc:302] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e375044d63649c097759c4b667eff79, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:08:59.722309 25589 raft_consensus.cc:2781] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:59.722348 25589 raft_consensus.cc:482] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:59.722373 25589 raft_consensus.cc:3037] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.791589 25834 raft_consensus.cc:482] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:59.791735 25834 raft_consensus.cc:504] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:59.792228 25834 leader_election.cc:288] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343)
I20240501 14:08:59.792691 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a1abcad3534f668299a245393d9022" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865" is_pre_election: true
I20240501 14:08:59.792690 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a1abcad3534f668299a245393d9022" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6" is_pre_election: true
I20240501 14:08:59.792852 25163 raft_consensus.cc:2454] T 78a1abcad3534f668299a245393d9022 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 0.
I20240501 14:08:59.792887 24897 raft_consensus.cc:2454] T 78a1abcad3534f668299a245393d9022 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 0.
I20240501 14:08:59.793190 24569 leader_election.cc:302] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:59.793407 25834 raft_consensus.cc:2781] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:59.793462 25834 raft_consensus.cc:482] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:59.793495 25834 raft_consensus.cc:3037] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.812498 25834 raft_consensus.cc:504] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:59.812498 25589 raft_consensus.cc:504] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.812974 25834 leader_election.cc:288] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 election: Requested vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343)
I20240501 14:08:59.812969 25589 leader_election.cc:288] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Requested vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:08:59.813419 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a1abcad3534f668299a245393d9022" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865"
I20240501 14:08:59.813555 25163 raft_consensus.cc:3037] T 78a1abcad3534f668299a245393d9022 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.813568 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b311d8b2b3f40f88c1580e9c24fcd4b" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:08:59.813621 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b311d8b2b3f40f88c1580e9c24fcd4b" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
I20240501 14:08:59.813681 24501 raft_consensus.cc:3037] T 7b311d8b2b3f40f88c1580e9c24fcd4b P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.813750 24765 raft_consensus.cc:3037] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.814543 24501 raft_consensus.cc:2454] T 7b311d8b2b3f40f88c1580e9c24fcd4b P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:08:59.814543 25163 raft_consensus.cc:2454] T 78a1abcad3534f668299a245393d9022 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 1.
I20240501 14:08:59.814724 24765 raft_consensus.cc:2454] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:08:59.814872 24569 leader_election.cc:302] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:08:59.814872 24964 leader_election.cc:302] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e375044d63649c097759c4b667eff79, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:08:59.815047 25589 raft_consensus.cc:2781] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:59.815116 25589 raft_consensus.cc:686] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [term 1 LEADER]: Becoming Leader. State: Replica: 7e375044d63649c097759c4b667eff79, State: Running, Role: LEADER
I20240501 14:08:59.815237 25834 raft_consensus.cc:2781] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:59.815236 25589 consensus_queue.cc:237] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:08:59.815315 25834 raft_consensus.cc:686] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 LEADER]: Becoming Leader. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Running, Role: LEADER
I20240501 14:08:59.815428 25834 consensus_queue.cc:237] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:08:59.815559 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a1abcad3534f668299a245393d9022" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6"
I20240501 14:08:59.815678 24897 raft_consensus.cc:3037] T 78a1abcad3534f668299a245393d9022 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.816570 24897 raft_consensus.cc:2454] T 78a1abcad3534f668299a245393d9022 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 1.
I20240501 14:08:59.818280 24353 catalog_manager.cc:5484] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66). New cstate: current_term: 1 leader_uuid: "f68136de04c94e5cb4a5f99881f02ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:59.818280 24359 catalog_manager.cc:5484] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e375044d63649c097759c4b667eff79 (127.23.191.69). New cstate: current_term: 1 leader_uuid: "7e375044d63649c097759c4b667eff79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:00.027410 25569 raft_consensus.cc:482] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:00.027542 25569 raft_consensus.cc:504] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:09:00.027925 25569 leader_election.cc:288] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343), f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:09:00.028385 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071272f7e339410890c23fb7e77465c1" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3" is_pre_election: true
I20240501 14:09:00.028545 24634 raft_consensus.cc:2454] T 071272f7e339410890c23fb7e77465c1 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 0.
I20240501 14:09:00.028512 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071272f7e339410890c23fb7e77465c1" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865" is_pre_election: true
I20240501 14:09:00.028648 25163 raft_consensus.cc:2454] T 071272f7e339410890c23fb7e77465c1 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 0.
I20240501 14:09:00.028898 24831 leader_election.cc:302] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2691129fd82431bb867cb1307e009b6, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:09:00.029104 25569 raft_consensus.cc:2781] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:00.029153 25569 raft_consensus.cc:482] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:00.029191 25569 raft_consensus.cc:3037] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:00.109318 25569 raft_consensus.cc:504] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:09:00.109820 25569 leader_election.cc:288] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 election: Requested vote from peers 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343), f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:09:00.110319 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071272f7e339410890c23fb7e77465c1" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3"
I20240501 14:09:00.110324 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071272f7e339410890c23fb7e77465c1" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865"
I20240501 14:09:00.110466 25163 raft_consensus.cc:3037] T 071272f7e339410890c23fb7e77465c1 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:00.110466 24634 raft_consensus.cc:3037] T 071272f7e339410890c23fb7e77465c1 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:00.111439 25163 raft_consensus.cc:2454] T 071272f7e339410890c23fb7e77465c1 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 1.
I20240501 14:09:00.111439 24634 raft_consensus.cc:2454] T 071272f7e339410890c23fb7e77465c1 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 1.
I20240501 14:09:00.111809 24831 leader_election.cc:302] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2691129fd82431bb867cb1307e009b6, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:09:00.112039 25569 raft_consensus.cc:2781] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:00.112229 25569 raft_consensus.cc:686] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [term 1 LEADER]: Becoming Leader. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Running, Role: LEADER
I20240501 14:09:00.112388 25569 consensus_queue.cc:237] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:09:00.114254 24353 catalog_manager.cc:5484] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2691129fd82431bb867cb1307e009b6 (127.23.191.68). New cstate: current_term: 1 leader_uuid: "c2691129fd82431bb867cb1307e009b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:00.236939 25834 consensus_queue.cc:1035] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:00.240300 25861 consensus_queue.cc:1035] T 78a1abcad3534f668299a245393d9022 P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:00.291937 25817 consensus_queue.cc:1035] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:00.295748 25817 consensus_queue.cc:1035] T 7b311d8b2b3f40f88c1580e9c24fcd4b P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:00.337612 25858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:00.337873 25858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:00.340925 25858 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 14:09:00.497551 24634 raft_consensus.cc:1261] T 071272f7e339410890c23fb7e77465c1 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Refusing update from remote peer c2691129fd82431bb867cb1307e009b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:00.498157 25569 consensus_queue.cc:1035] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:00.499008 25163 raft_consensus.cc:1261] T 071272f7e339410890c23fb7e77465c1 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Refusing update from remote peer c2691129fd82431bb867cb1307e009b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:00.499423 25569 consensus_queue.cc:1035] T 071272f7e339410890c23fb7e77465c1 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 154.508 ms
time per row: 0.077254 ms
I20240501 14:09:00.542346 24353 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43068:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:00.548400 24482 tablet_service.cc:1467] Processing CreateTablet for tablet 8023325ac74d43faa0817b49d3ef9707 (DEFAULT_TABLE table=rebalance_test_table_8 [id=90ea7dd3686d436f88bfdf2bb69a04c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.548597 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 8023325ac74d43faa0817b49d3ef9707 (DEFAULT_TABLE table=rebalance_test_table_8 [id=90ea7dd3686d436f88bfdf2bb69a04c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.548730 24482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8023325ac74d43faa0817b49d3ef9707. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.548664 24613 tablet_service.cc:1467] Processing CreateTablet for tablet 1d557db356fa42afa39e059f0b56ee20 (DEFAULT_TABLE table=rebalance_test_table_8 [id=90ea7dd3686d436f88bfdf2bb69a04c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.548914 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8023325ac74d43faa0817b49d3ef9707. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.548923 25009 tablet_service.cc:1467] Processing CreateTablet for tablet 8023325ac74d43faa0817b49d3ef9707 (DEFAULT_TABLE table=rebalance_test_table_8 [id=90ea7dd3686d436f88bfdf2bb69a04c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.549156 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8023325ac74d43faa0817b49d3ef9707. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.549170 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d557db356fa42afa39e059f0b56ee20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.551398 25143 tablet_service.cc:1467] Processing CreateTablet for tablet 1d557db356fa42afa39e059f0b56ee20 (DEFAULT_TABLE table=rebalance_test_table_8 [id=90ea7dd3686d436f88bfdf2bb69a04c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.551496 25922 tablet_bootstrap.cc:492] T 8023325ac74d43faa0817b49d3ef9707 P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:09:00.551698 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d557db356fa42afa39e059f0b56ee20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.552119 24878 tablet_service.cc:1467] Processing CreateTablet for tablet 1d557db356fa42afa39e059f0b56ee20 (DEFAULT_TABLE table=rebalance_test_table_8 [id=90ea7dd3686d436f88bfdf2bb69a04c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.552464 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d557db356fa42afa39e059f0b56ee20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.554824 24482 tablet_service.cc:1467] Processing CreateTablet for tablet 05b3283055ad40149a175eebfa58ee3c (DEFAULT_TABLE table=rebalance_test_table_8 [id=90ea7dd3686d436f88bfdf2bb69a04c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.554912 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 05b3283055ad40149a175eebfa58ee3c (DEFAULT_TABLE table=rebalance_test_table_8 [id=90ea7dd3686d436f88bfdf2bb69a04c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.554900 25009 tablet_service.cc:1467] Processing CreateTablet for tablet 05b3283055ad40149a175eebfa58ee3c (DEFAULT_TABLE table=rebalance_test_table_8 [id=90ea7dd3686d436f88bfdf2bb69a04c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.555037 25923 tablet_bootstrap.cc:492] T 1d557db356fa42afa39e059f0b56ee20 P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:09:00.555078 24482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b3283055ad40149a175eebfa58ee3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.555126 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b3283055ad40149a175eebfa58ee3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.555137 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b3283055ad40149a175eebfa58ee3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.555974 25923 tablet_bootstrap.cc:654] T 1d557db356fa42afa39e059f0b56ee20 P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.557116 25924 tablet_bootstrap.cc:492] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:09:00.558077 25922 tablet_bootstrap.cc:654] T 8023325ac74d43faa0817b49d3ef9707 P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.558077 25924 tablet_bootstrap.cc:654] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.558913 25920 tablet_bootstrap.cc:492] T 8023325ac74d43faa0817b49d3ef9707 P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:09:00.559340 25922 tablet_bootstrap.cc:492] T 8023325ac74d43faa0817b49d3ef9707 P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:09:00.559504 25922 ts_tablet_manager.cc:1367] T 8023325ac74d43faa0817b49d3ef9707 P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:00.559852 25920 tablet_bootstrap.cc:654] T 8023325ac74d43faa0817b49d3ef9707 P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.560063 25922 raft_consensus.cc:348] T 8023325ac74d43faa0817b49d3ef9707 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:00.560200 25922 raft_consensus.cc:374] T 8023325ac74d43faa0817b49d3ef9707 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.560235 25922 raft_consensus.cc:729] T 8023325ac74d43faa0817b49d3ef9707 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.560437 25922 consensus_queue.cc:260] T 8023325ac74d43faa0817b49d3ef9707 P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:00.560734 25922 ts_tablet_manager.cc:1398] T 8023325ac74d43faa0817b49d3ef9707 P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:00.561007 25925 tablet_bootstrap.cc:492] T 1d557db356fa42afa39e059f0b56ee20 P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:09:00.558768 25921 tablet_bootstrap.cc:492] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:09:00.561390 25922 tablet_bootstrap.cc:492] T 05b3283055ad40149a175eebfa58ee3c P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:09:00.561502 25923 tablet_bootstrap.cc:492] T 1d557db356fa42afa39e059f0b56ee20 P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:09:00.561605 25923 ts_tablet_manager.cc:1367] T 1d557db356fa42afa39e059f0b56ee20 P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:00.562093 25923 raft_consensus.cc:348] T 1d557db356fa42afa39e059f0b56ee20 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.562220 25923 raft_consensus.cc:374] T 1d557db356fa42afa39e059f0b56ee20 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.562251 25923 raft_consensus.cc:729] T 1d557db356fa42afa39e059f0b56ee20 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.562286 25922 tablet_bootstrap.cc:654] T 05b3283055ad40149a175eebfa58ee3c P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.562391 25923 consensus_queue.cc:260] T 1d557db356fa42afa39e059f0b56ee20 P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.562671 25923 ts_tablet_manager.cc:1398] T 1d557db356fa42afa39e059f0b56ee20 P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:00.562772 25925 tablet_bootstrap.cc:654] T 1d557db356fa42afa39e059f0b56ee20 P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.563477 25924 tablet_bootstrap.cc:492] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:09:00.563570 25924 ts_tablet_manager.cc:1367] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:00.564342 25924 raft_consensus.cc:348] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.564469 25924 raft_consensus.cc:374] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.564500 25924 raft_consensus.cc:729] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.564618 25924 consensus_queue.cc:260] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.564940 25924 ts_tablet_manager.cc:1398] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:00.565140 25921 tablet_bootstrap.cc:654] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.565932 25920 tablet_bootstrap.cc:492] T 8023325ac74d43faa0817b49d3ef9707 P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:09:00.566025 25920 ts_tablet_manager.cc:1367] T 8023325ac74d43faa0817b49d3ef9707 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:00.567030 25920 raft_consensus.cc:348] T 8023325ac74d43faa0817b49d3ef9707 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:00.567160 25922 tablet_bootstrap.cc:492] T 05b3283055ad40149a175eebfa58ee3c P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:09:00.567160 25920 raft_consensus.cc:374] T 8023325ac74d43faa0817b49d3ef9707 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.567196 25920 raft_consensus.cc:729] T 8023325ac74d43faa0817b49d3ef9707 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.567238 25922 ts_tablet_manager.cc:1367] T 05b3283055ad40149a175eebfa58ee3c P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:00.567320 25920 consensus_queue.cc:260] T 8023325ac74d43faa0817b49d3ef9707 P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:00.567646 25922 raft_consensus.cc:348] T 05b3283055ad40149a175eebfa58ee3c P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:00.568065 25920 ts_tablet_manager.cc:1398] T 8023325ac74d43faa0817b49d3ef9707 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:00.568260 25920 tablet_bootstrap.cc:492] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:09:00.568966 25922 raft_consensus.cc:374] T 05b3283055ad40149a175eebfa58ee3c P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.569022 25922 raft_consensus.cc:729] T 05b3283055ad40149a175eebfa58ee3c P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.569340 25920 tablet_bootstrap.cc:654] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.570003 25922 consensus_queue.cc:260] T 05b3283055ad40149a175eebfa58ee3c P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:00.570322 25922 ts_tablet_manager.cc:1398] T 05b3283055ad40149a175eebfa58ee3c P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:00.571295 25925 tablet_bootstrap.cc:492] T 1d557db356fa42afa39e059f0b56ee20 P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:09:00.571974 25920 tablet_bootstrap.cc:492] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:09:00.572108 25920 ts_tablet_manager.cc:1367] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:09:00.572744 25920 raft_consensus.cc:348] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:00.572885 25920 raft_consensus.cc:374] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.572921 25920 raft_consensus.cc:729] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.572858 25925 ts_tablet_manager.cc:1367] T 1d557db356fa42afa39e059f0b56ee20 P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:00.573038 25920 consensus_queue.cc:260] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:00.573413 25920 ts_tablet_manager.cc:1398] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:00.573495 25925 raft_consensus.cc:348] T 1d557db356fa42afa39e059f0b56ee20 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.573594 25925 raft_consensus.cc:374] T 1d557db356fa42afa39e059f0b56ee20 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.573617 25925 raft_consensus.cc:729] T 1d557db356fa42afa39e059f0b56ee20 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.573726 25925 consensus_queue.cc:260] T 1d557db356fa42afa39e059f0b56ee20 P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.574010 25925 ts_tablet_manager.cc:1398] T 1d557db356fa42afa39e059f0b56ee20 P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:00.575572 25921 tablet_bootstrap.cc:492] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:09:00.576238 25921 ts_tablet_manager.cc:1367] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 14:09:00.577358 25921 raft_consensus.cc:348] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:00.577453 25921 raft_consensus.cc:374] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.577474 25921 raft_consensus.cc:729] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.577585 25921 consensus_queue.cc:260] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:00.577836 25921 ts_tablet_manager.cc:1398] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:00.577982 25921 tablet_bootstrap.cc:492] T 05b3283055ad40149a175eebfa58ee3c P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:09:00.578917 25921 tablet_bootstrap.cc:654] T 05b3283055ad40149a175eebfa58ee3c P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.585275 25921 tablet_bootstrap.cc:492] T 05b3283055ad40149a175eebfa58ee3c P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:09:00.585371 25921 ts_tablet_manager.cc:1367] T 05b3283055ad40149a175eebfa58ee3c P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:00.585865 25921 raft_consensus.cc:348] T 05b3283055ad40149a175eebfa58ee3c P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:00.585992 25921 raft_consensus.cc:374] T 05b3283055ad40149a175eebfa58ee3c P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.586278 25921 raft_consensus.cc:729] T 05b3283055ad40149a175eebfa58ee3c P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.586428 25921 consensus_queue.cc:260] T 05b3283055ad40149a175eebfa58ee3c P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:00.586791 25921 ts_tablet_manager.cc:1398] T 05b3283055ad40149a175eebfa58ee3c P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:00.639977 25866 raft_consensus.cc:482] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:00.640112 25866 raft_consensus.cc:504] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:00.641294 25866 leader_election.cc:288] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311), 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635)
I20240501 14:09:00.645359 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b3283055ad40149a175eebfa58ee3c" candidate_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" is_pre_election: true
I20240501 14:09:00.645507 24765 raft_consensus.cc:2454] T 05b3283055ad40149a175eebfa58ee3c P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd8ec7cff3a147e5adfc4eb329a23895 in term 0.
I20240501 14:09:00.645884 24435 leader_election.cc:302] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:09:00.646116 25866 raft_consensus.cc:2781] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:00.646173 25866 raft_consensus.cc:482] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:00.646210 25866 raft_consensus.cc:3037] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:00.646921 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b3283055ad40149a175eebfa58ee3c" candidate_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79" is_pre_election: true
I20240501 14:09:00.647076 25029 raft_consensus.cc:2454] T 05b3283055ad40149a175eebfa58ee3c P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd8ec7cff3a147e5adfc4eb329a23895 in term 0.
I20240501 14:09:00.647215 25866 raft_consensus.cc:504] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:00.647558 25866 leader_election.cc:288] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [CANDIDATE]: Term 1 election: Requested vote from peers 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311), 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635)
I20240501 14:09:00.647878 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b3283055ad40149a175eebfa58ee3c" candidate_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
I20240501 14:09:00.647998 24765 raft_consensus.cc:3037] T 05b3283055ad40149a175eebfa58ee3c P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:00.647994 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b3283055ad40149a175eebfa58ee3c" candidate_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79"
I20240501 14:09:00.648109 25029 raft_consensus.cc:3037] T 05b3283055ad40149a175eebfa58ee3c P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:00.648741 24765 raft_consensus.cc:2454] T 05b3283055ad40149a175eebfa58ee3c P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd8ec7cff3a147e5adfc4eb329a23895 in term 1.
I20240501 14:09:00.649009 25029 raft_consensus.cc:2454] T 05b3283055ad40149a175eebfa58ee3c P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd8ec7cff3a147e5adfc4eb329a23895 in term 1.
I20240501 14:09:00.649032 24435 leader_election.cc:302] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:09:00.649333 25866 raft_consensus.cc:2781] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:00.649542 25866 raft_consensus.cc:686] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 LEADER]: Becoming Leader. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Running, Role: LEADER
I20240501 14:09:00.649850 25866 consensus_queue.cc:237] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:00.652590 24360 catalog_manager.cc:5484] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:00.669919 25263 raft_consensus.cc:482] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:00.670037 25263 raft_consensus.cc:504] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.670444 25263 leader_election.cc:288] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345), c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971)
I20240501 14:09:00.670809 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d557db356fa42afa39e059f0b56ee20" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3" is_pre_election: true
I20240501 14:09:00.670867 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d557db356fa42afa39e059f0b56ee20" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6" is_pre_election: true
I20240501 14:09:00.670941 24634 raft_consensus.cc:2454] T 1d557db356fa42afa39e059f0b56ee20 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:09:00.671010 24897 raft_consensus.cc:2454] T 1d557db356fa42afa39e059f0b56ee20 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:09:00.671121 25880 raft_consensus.cc:482] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:00.671245 25095 leader_election.cc:302] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:09:00.671223 25880 raft_consensus.cc:504] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:00.671427 25263 raft_consensus.cc:2781] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:00.671484 25263 raft_consensus.cc:482] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:00.671530 25263 raft_consensus.cc:3037] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:00.671646 25880 leader_election.cc:288] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:09:00.671957 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8023325ac74d43faa0817b49d3ef9707" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79" is_pre_election: true
I20240501 14:09:00.672078 25029 raft_consensus.cc:2454] T 8023325ac74d43faa0817b49d3ef9707 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 0.
I20240501 14:09:00.672138 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8023325ac74d43faa0817b49d3ef9707" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:09:00.672297 24501 raft_consensus.cc:2454] T 8023325ac74d43faa0817b49d3ef9707 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 0.
I20240501 14:09:00.672528 24699 leader_election.cc:302] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:09:00.672523 25263 raft_consensus.cc:504] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.672734 25880 raft_consensus.cc:2781] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:00.672780 25880 raft_consensus.cc:482] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:00.672804 25880 raft_consensus.cc:3037] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:00.672904 25263 leader_election.cc:288] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Requested vote from peers f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345), c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971)
I20240501 14:09:00.673244 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d557db356fa42afa39e059f0b56ee20" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3"
I20240501 14:09:00.673254 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d557db356fa42afa39e059f0b56ee20" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6"
I20240501 14:09:00.673372 24897 raft_consensus.cc:3037] T 1d557db356fa42afa39e059f0b56ee20 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:00.673372 24634 raft_consensus.cc:3037] T 1d557db356fa42afa39e059f0b56ee20 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:00.673749 25880 raft_consensus.cc:504] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:00.674062 25880 leader_election.cc:288] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 election: Requested vote from peers 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:09:00.674239 24897 raft_consensus.cc:2454] T 1d557db356fa42afa39e059f0b56ee20 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:09:00.674239 24634 raft_consensus.cc:2454] T 1d557db356fa42afa39e059f0b56ee20 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:09:00.674437 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8023325ac74d43faa0817b49d3ef9707" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79"
I20240501 14:09:00.674551 25029 raft_consensus.cc:3037] T 8023325ac74d43faa0817b49d3ef9707 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:00.674537 25098 leader_election.cc:302] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, c2691129fd82431bb867cb1307e009b6; no voters:
I20240501 14:09:00.674779 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8023325ac74d43faa0817b49d3ef9707" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:09:00.674872 24501 raft_consensus.cc:3037] T 8023325ac74d43faa0817b49d3ef9707 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:00.675053 25263 raft_consensus.cc:2781] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:00.675143 25263 raft_consensus.cc:686] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [term 1 LEADER]: Becoming Leader. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Running, Role: LEADER
I20240501 14:09:00.675264 25263 consensus_queue.cc:237] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.675405 25029 raft_consensus.cc:2454] T 8023325ac74d43faa0817b49d3ef9707 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 1.
I20240501 14:09:00.675644 24699 leader_election.cc:302] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:09:00.675796 25880 raft_consensus.cc:2781] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:00.675884 25880 raft_consensus.cc:686] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 LEADER]: Becoming Leader. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Running, Role: LEADER
I20240501 14:09:00.675997 25880 consensus_queue.cc:237] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:00.677672 24351 catalog_manager.cc:5484] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67). New cstate: current_term: 1 leader_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:00.677711 24360 catalog_manager.cc:5484] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70). New cstate: current_term: 1 leader_uuid: "786f81b7c57f452e9b463f721fc3e865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:00.697532 24501 raft_consensus.cc:2454] T 8023325ac74d43faa0817b49d3ef9707 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 1.
W20240501 14:09:00.792968 25943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:00.793218 25943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:00.797015 25943 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 14:09:00.880290 24501 raft_consensus.cc:1261] T 8023325ac74d43faa0817b49d3ef9707 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Refusing update from remote peer 319fbe37b2ce4434a8d2fe879ca920a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:00.880678 25029 raft_consensus.cc:1261] T 8023325ac74d43faa0817b49d3ef9707 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Refusing update from remote peer 319fbe37b2ce4434a8d2fe879ca920a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:00.881122 25880 consensus_queue.cc:1035] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:00.882575 25880 consensus_queue.cc:1035] T 8023325ac74d43faa0817b49d3ef9707 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:00.889184 24897 raft_consensus.cc:1261] T 1d557db356fa42afa39e059f0b56ee20 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:00.889770 25263 consensus_queue.cc:1035] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:00.890514 24634 raft_consensus.cc:1261] T 1d557db356fa42afa39e059f0b56ee20 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:00.891135 25883 consensus_queue.cc:1035] T 1d557db356fa42afa39e059f0b56ee20 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:00.908037 24765 raft_consensus.cc:1261] T 05b3283055ad40149a175eebfa58ee3c P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Refusing update from remote peer fd8ec7cff3a147e5adfc4eb329a23895: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:00.908279 25030 raft_consensus.cc:1261] T 05b3283055ad40149a175eebfa58ee3c P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Refusing update from remote peer fd8ec7cff3a147e5adfc4eb329a23895: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:00.908492 25866 consensus_queue.cc:1035] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:00.908663 25860 mvcc.cc:205] Tried to move back new op lower bound from 7022889127449333760 to 7022889126607052800. Current Snapshot: MvccSnapshot[applied={T|T < 7022889127449333760}]
I20240501 14:09:00.909271 25938 consensus_queue.cc:1035] T 05b3283055ad40149a175eebfa58ee3c P fd8ec7cff3a147e5adfc4eb329a23895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:00.940135 25972 mvcc.cc:205] Tried to move back new op lower bound from 7022889127547334656 to 7022889126503997440. Current Snapshot: MvccSnapshot[applied={T|T < 7022889127547334656}]
I20240501 14:09:00.940874 25871 mvcc.cc:205] Tried to move back new op lower bound from 7022889127547334656 to 7022889126503997440. Current Snapshot: MvccSnapshot[applied={T|T < 7022889127478857728}]
I20240501 14:09:00.940654 25982 mvcc.cc:205] Tried to move back new op lower bound from 7022889127547334656 to 7022889126503997440. Current Snapshot: MvccSnapshot[applied={T|T < 7022889127478857728}]
INSERT report
rows total: 2000
time total: 120.506 ms
time per row: 0.060253 ms
I20240501 14:09:00.959575 24353 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43068:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:00.965119 24878 tablet_service.cc:1467] Processing CreateTablet for tablet 4603c0d1e9fa433d8e03f6a4dadad53a (DEFAULT_TABLE table=rebalance_test_table_9 [id=bfdc5165fcfe48eea1b6e38523090b05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.965140 24613 tablet_service.cc:1467] Processing CreateTablet for tablet 4603c0d1e9fa433d8e03f6a4dadad53a (DEFAULT_TABLE table=rebalance_test_table_9 [id=bfdc5165fcfe48eea1b6e38523090b05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.965468 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4603c0d1e9fa433d8e03f6a4dadad53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.965468 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4603c0d1e9fa433d8e03f6a4dadad53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.966017 24745 tablet_service.cc:1467] Processing CreateTablet for tablet e889bdce41804ba0a096a66c34c7c8f1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=bfdc5165fcfe48eea1b6e38523090b05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.966128 24482 tablet_service.cc:1467] Processing CreateTablet for tablet e889bdce41804ba0a096a66c34c7c8f1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=bfdc5165fcfe48eea1b6e38523090b05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.966264 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e889bdce41804ba0a096a66c34c7c8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.966396 24482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e889bdce41804ba0a096a66c34c7c8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.967605 25925 tablet_bootstrap.cc:492] T 4603c0d1e9fa433d8e03f6a4dadad53a P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:09:00.967629 25923 tablet_bootstrap.cc:492] T 4603c0d1e9fa433d8e03f6a4dadad53a P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:09:00.968350 25921 tablet_bootstrap.cc:492] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:09:00.968492 25923 tablet_bootstrap.cc:654] T 4603c0d1e9fa433d8e03f6a4dadad53a P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.968711 25925 tablet_bootstrap.cc:654] T 4603c0d1e9fa433d8e03f6a4dadad53a P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.969019 25920 tablet_bootstrap.cc:492] T e889bdce41804ba0a096a66c34c7c8f1 P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:09:00.969158 25921 tablet_bootstrap.cc:654] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.969811 25920 tablet_bootstrap.cc:654] T e889bdce41804ba0a096a66c34c7c8f1 P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.970134 25923 tablet_bootstrap.cc:492] T 4603c0d1e9fa433d8e03f6a4dadad53a P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:09:00.970212 25923 ts_tablet_manager.cc:1367] T 4603c0d1e9fa433d8e03f6a4dadad53a P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:00.970284 25009 tablet_service.cc:1467] Processing CreateTablet for tablet e889bdce41804ba0a096a66c34c7c8f1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=bfdc5165fcfe48eea1b6e38523090b05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.970535 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e889bdce41804ba0a096a66c34c7c8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.970624 25923 raft_consensus.cc:348] T 4603c0d1e9fa433d8e03f6a4dadad53a P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.970721 25923 raft_consensus.cc:374] T 4603c0d1e9fa433d8e03f6a4dadad53a P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.970752 25923 raft_consensus.cc:729] T 4603c0d1e9fa433d8e03f6a4dadad53a P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.970870 25923 consensus_queue.cc:260] T 4603c0d1e9fa433d8e03f6a4dadad53a P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.971001 25142 tablet_service.cc:1467] Processing CreateTablet for tablet 6b90a26d65724b7e9be211b84b5da7fa (DEFAULT_TABLE table=rebalance_test_table_9 [id=bfdc5165fcfe48eea1b6e38523090b05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.971122 25923 ts_tablet_manager.cc:1398] T 4603c0d1e9fa433d8e03f6a4dadad53a P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:00.971274 25142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b90a26d65724b7e9be211b84b5da7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.971688 24613 tablet_service.cc:1467] Processing CreateTablet for tablet 6b90a26d65724b7e9be211b84b5da7fa (DEFAULT_TABLE table=rebalance_test_table_9 [id=bfdc5165fcfe48eea1b6e38523090b05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.972268 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b90a26d65724b7e9be211b84b5da7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.972759 25143 tablet_service.cc:1467] Processing CreateTablet for tablet 4603c0d1e9fa433d8e03f6a4dadad53a (DEFAULT_TABLE table=rebalance_test_table_9 [id=bfdc5165fcfe48eea1b6e38523090b05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.973011 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4603c0d1e9fa433d8e03f6a4dadad53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.974368 24878 tablet_service.cc:1467] Processing CreateTablet for tablet 6b90a26d65724b7e9be211b84b5da7fa (DEFAULT_TABLE table=rebalance_test_table_9 [id=bfdc5165fcfe48eea1b6e38523090b05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:00.974623 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b90a26d65724b7e9be211b84b5da7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.974753 25920 tablet_bootstrap.cc:492] T e889bdce41804ba0a096a66c34c7c8f1 P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:09:00.974840 25920 ts_tablet_manager.cc:1367] T e889bdce41804ba0a096a66c34c7c8f1 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 14:09:00.974942 25922 tablet_bootstrap.cc:492] T e889bdce41804ba0a096a66c34c7c8f1 P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:09:00.975292 25920 raft_consensus.cc:348] T e889bdce41804ba0a096a66c34c7c8f1 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:00.975385 25920 raft_consensus.cc:374] T e889bdce41804ba0a096a66c34c7c8f1 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.975414 25920 raft_consensus.cc:729] T e889bdce41804ba0a096a66c34c7c8f1 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.975461 25923 tablet_bootstrap.cc:492] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:09:00.975533 25920 consensus_queue.cc:260] T e889bdce41804ba0a096a66c34c7c8f1 P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:00.975764 25922 tablet_bootstrap.cc:654] T e889bdce41804ba0a096a66c34c7c8f1 P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.975813 25920 ts_tablet_manager.cc:1398] T e889bdce41804ba0a096a66c34c7c8f1 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:09:00.976284 25923 tablet_bootstrap.cc:654] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.976364 25924 tablet_bootstrap.cc:492] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:09:00.977267 25924 tablet_bootstrap.cc:654] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.978121 25921 tablet_bootstrap.cc:492] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:09:00.978206 25921 ts_tablet_manager.cc:1367] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:09:00.978550 25922 tablet_bootstrap.cc:492] T e889bdce41804ba0a096a66c34c7c8f1 P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:09:00.978632 25922 ts_tablet_manager.cc:1367] T e889bdce41804ba0a096a66c34c7c8f1 P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:00.978706 25921 raft_consensus.cc:348] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:00.978818 25921 raft_consensus.cc:374] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.978850 25921 raft_consensus.cc:729] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.978976 25921 consensus_queue.cc:260] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:00.979007 25922 raft_consensus.cc:348] T e889bdce41804ba0a096a66c34c7c8f1 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:00.979116 25922 raft_consensus.cc:374] T e889bdce41804ba0a096a66c34c7c8f1 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.979146 25922 raft_consensus.cc:729] T e889bdce41804ba0a096a66c34c7c8f1 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.979262 25921 ts_tablet_manager.cc:1398] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:00.979255 25922 consensus_queue.cc:260] T e889bdce41804ba0a096a66c34c7c8f1 P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:00.979511 25922 ts_tablet_manager.cc:1398] T e889bdce41804ba0a096a66c34c7c8f1 P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:00.979830 25924 tablet_bootstrap.cc:492] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:09:00.979910 25924 ts_tablet_manager.cc:1367] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:00.980304 25924 raft_consensus.cc:348] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.980398 25924 raft_consensus.cc:374] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.980427 25924 raft_consensus.cc:729] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.980535 25924 consensus_queue.cc:260] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.980774 25924 ts_tablet_manager.cc:1398] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:00.981071 25924 tablet_bootstrap.cc:492] T 6b90a26d65724b7e9be211b84b5da7fa P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:09:00.982208 25924 tablet_bootstrap.cc:654] T 6b90a26d65724b7e9be211b84b5da7fa P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.983601 25924 tablet_bootstrap.cc:492] T 6b90a26d65724b7e9be211b84b5da7fa P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:09:00.983695 25924 ts_tablet_manager.cc:1367] T 6b90a26d65724b7e9be211b84b5da7fa P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:00.984104 25924 raft_consensus.cc:348] T 6b90a26d65724b7e9be211b84b5da7fa P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:09:00.984200 25924 raft_consensus.cc:374] T 6b90a26d65724b7e9be211b84b5da7fa P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.984230 25924 raft_consensus.cc:729] T 6b90a26d65724b7e9be211b84b5da7fa P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.984344 25924 consensus_queue.cc:260] T 6b90a26d65724b7e9be211b84b5da7fa P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:09:00.984776 25924 ts_tablet_manager.cc:1398] T 6b90a26d65724b7e9be211b84b5da7fa P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:00.989281 25925 tablet_bootstrap.cc:492] T 4603c0d1e9fa433d8e03f6a4dadad53a P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:09:00.989388 25925 ts_tablet_manager.cc:1367] T 4603c0d1e9fa433d8e03f6a4dadad53a P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:09:00.989742 25925 raft_consensus.cc:348] T 4603c0d1e9fa433d8e03f6a4dadad53a P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.989848 25925 raft_consensus.cc:374] T 4603c0d1e9fa433d8e03f6a4dadad53a P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.989878 25925 raft_consensus.cc:729] T 4603c0d1e9fa433d8e03f6a4dadad53a P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.990020 25925 consensus_queue.cc:260] T 4603c0d1e9fa433d8e03f6a4dadad53a P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:00.990289 25925 ts_tablet_manager.cc:1398] T 4603c0d1e9fa433d8e03f6a4dadad53a P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:00.990465 25925 tablet_bootstrap.cc:492] T 6b90a26d65724b7e9be211b84b5da7fa P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:09:00.991375 25925 tablet_bootstrap.cc:654] T 6b90a26d65724b7e9be211b84b5da7fa P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:00.992393 25925 tablet_bootstrap.cc:492] T 6b90a26d65724b7e9be211b84b5da7fa P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:09:00.992489 25925 ts_tablet_manager.cc:1367] T 6b90a26d65724b7e9be211b84b5da7fa P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:00.993333 25925 raft_consensus.cc:348] T 6b90a26d65724b7e9be211b84b5da7fa P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:09:00.993444 25925 raft_consensus.cc:374] T 6b90a26d65724b7e9be211b84b5da7fa P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.993469 25925 raft_consensus.cc:729] T 6b90a26d65724b7e9be211b84b5da7fa P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.993629 25925 consensus_queue.cc:260] T 6b90a26d65724b7e9be211b84b5da7fa P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:09:00.993871 25925 ts_tablet_manager.cc:1398] T 6b90a26d65724b7e9be211b84b5da7fa P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:00.997205 25923 tablet_bootstrap.cc:492] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:09:00.997300 25923 ts_tablet_manager.cc:1367] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:09:00.997684 25923 raft_consensus.cc:348] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:09:00.997805 25923 raft_consensus.cc:374] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:00.997848 25923 raft_consensus.cc:729] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.997968 25923 consensus_queue.cc:260] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:09:00.998241 25923 ts_tablet_manager.cc:1398] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:01.092679 25883 raft_consensus.cc:482] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:01.092797 25883 raft_consensus.cc:504] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.093160 25883 leader_election.cc:288] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:09:01.093541 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4603c0d1e9fa433d8e03f6a4dadad53a" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3" is_pre_election: true
I20240501 14:09:01.093561 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4603c0d1e9fa433d8e03f6a4dadad53a" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6" is_pre_election: true
I20240501 14:09:01.093691 24634 raft_consensus.cc:2454] T 4603c0d1e9fa433d8e03f6a4dadad53a P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:09:01.093713 24897 raft_consensus.cc:2454] T 4603c0d1e9fa433d8e03f6a4dadad53a P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:09:01.094008 25095 leader_election.cc:302] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:09:01.094206 25883 raft_consensus.cc:2781] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:01.094310 25883 raft_consensus.cc:482] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:01.094352 25883 raft_consensus.cc:3037] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.095186 25883 raft_consensus.cc:504] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.095500 25883 leader_election.cc:288] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Requested vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:09:01.095814 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4603c0d1e9fa433d8e03f6a4dadad53a" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3"
I20240501 14:09:01.095827 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4603c0d1e9fa433d8e03f6a4dadad53a" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6"
I20240501 14:09:01.095911 24634 raft_consensus.cc:3037] T 4603c0d1e9fa433d8e03f6a4dadad53a P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.095923 24897 raft_consensus.cc:3037] T 4603c0d1e9fa433d8e03f6a4dadad53a P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.096578 24634 raft_consensus.cc:2454] T 4603c0d1e9fa433d8e03f6a4dadad53a P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:09:01.096586 24897 raft_consensus.cc:2454] T 4603c0d1e9fa433d8e03f6a4dadad53a P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:09:01.096863 25095 leader_election.cc:302] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:09:01.097024 25883 raft_consensus.cc:2781] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:01.097112 25883 raft_consensus.cc:686] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [term 1 LEADER]: Becoming Leader. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Running, Role: LEADER
I20240501 14:09:01.097241 25883 consensus_queue.cc:237] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.099246 24353 catalog_manager.cc:5484] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70). New cstate: current_term: 1 leader_uuid: "786f81b7c57f452e9b463f721fc3e865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:01.100996 25979 raft_consensus.cc:482] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:01.101114 25979 raft_consensus.cc:504] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:01.101522 25979 leader_election.cc:288] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635)
I20240501 14:09:01.101945 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e889bdce41804ba0a096a66c34c7c8f1" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79" is_pre_election: true
I20240501 14:09:01.102016 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e889bdce41804ba0a096a66c34c7c8f1" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:09:01.102077 25030 raft_consensus.cc:2454] T e889bdce41804ba0a096a66c34c7c8f1 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 0.
I20240501 14:09:01.102169 24501 raft_consensus.cc:2454] T e889bdce41804ba0a096a66c34c7c8f1 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 0.
I20240501 14:09:01.102386 24699 leader_election.cc:302] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:09:01.102552 25979 raft_consensus.cc:2781] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:01.102613 25979 raft_consensus.cc:482] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:01.102658 25979 raft_consensus.cc:3037] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.103427 25979 raft_consensus.cc:504] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:01.103716 25979 leader_election.cc:288] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 election: Requested vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635)
I20240501 14:09:01.104028 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e889bdce41804ba0a096a66c34c7c8f1" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:09:01.104032 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e889bdce41804ba0a096a66c34c7c8f1" candidate_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79"
I20240501 14:09:01.104146 24501 raft_consensus.cc:3037] T e889bdce41804ba0a096a66c34c7c8f1 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.104146 25030 raft_consensus.cc:3037] T e889bdce41804ba0a096a66c34c7c8f1 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.105100 24501 raft_consensus.cc:2454] T e889bdce41804ba0a096a66c34c7c8f1 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 1.
I20240501 14:09:01.105110 25030 raft_consensus.cc:2454] T e889bdce41804ba0a096a66c34c7c8f1 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 319fbe37b2ce4434a8d2fe879ca920a7 in term 1.
I20240501 14:09:01.105407 24699 leader_election.cc:302] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:09:01.105557 25979 raft_consensus.cc:2781] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:01.105629 25979 raft_consensus.cc:686] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 LEADER]: Becoming Leader. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Running, Role: LEADER
I20240501 14:09:01.105782 25979 consensus_queue.cc:237] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:01.106531 25861 raft_consensus.cc:482] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:01.106613 25861 raft_consensus.cc:504] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:09:01.106958 25861 leader_election.cc:288] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343), c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971)
I20240501 14:09:01.107288 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b90a26d65724b7e9be211b84b5da7fa" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865" is_pre_election: true
I20240501 14:09:01.107388 25163 raft_consensus.cc:2454] T 6b90a26d65724b7e9be211b84b5da7fa P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 0.
I20240501 14:09:01.107439 24353 catalog_manager.cc:5484] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67). New cstate: current_term: 1 leader_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:01.107583 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b90a26d65724b7e9be211b84b5da7fa" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6" is_pre_election: true
I20240501 14:09:01.107676 24897 raft_consensus.cc:2454] T 6b90a26d65724b7e9be211b84b5da7fa P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 0.
I20240501 14:09:01.107676 24569 leader_election.cc:302] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:09:01.107831 25861 raft_consensus.cc:2781] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:01.107873 25861 raft_consensus.cc:482] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:01.107904 25861 raft_consensus.cc:3037] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.108611 25861 raft_consensus.cc:504] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:09:01.108939 25861 leader_election.cc:288] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 election: Requested vote from peers 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343), c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971)
I20240501 14:09:01.109243 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b90a26d65724b7e9be211b84b5da7fa" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865"
I20240501 14:09:01.109249 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b90a26d65724b7e9be211b84b5da7fa" candidate_uuid: "f68136de04c94e5cb4a5f99881f02ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6"
I20240501 14:09:01.109344 25163 raft_consensus.cc:3037] T 6b90a26d65724b7e9be211b84b5da7fa P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.109350 24897 raft_consensus.cc:3037] T 6b90a26d65724b7e9be211b84b5da7fa P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.109998 24897 raft_consensus.cc:2454] T 6b90a26d65724b7e9be211b84b5da7fa P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 1.
I20240501 14:09:01.110000 25163 raft_consensus.cc:2454] T 6b90a26d65724b7e9be211b84b5da7fa P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f68136de04c94e5cb4a5f99881f02ab3 in term 1.
I20240501 14:09:01.110502 24569 leader_election.cc:302] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:09:01.110929 25861 raft_consensus.cc:2781] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:01.111070 25861 raft_consensus.cc:686] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [term 1 LEADER]: Becoming Leader. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Running, Role: LEADER
I20240501 14:09:01.111196 25861 consensus_queue.cc:237] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } }
I20240501 14:09:01.112753 24353 catalog_manager.cc:5484] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66). New cstate: current_term: 1 leader_uuid: "f68136de04c94e5cb4a5f99881f02ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:01.235441 26001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:01.235675 26001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:01.238680 26001 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 14:09:01.322448 24634 raft_consensus.cc:1261] T 4603c0d1e9fa433d8e03f6a4dadad53a P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:01.323582 25883 consensus_queue.cc:1035] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:01.324158 24897 raft_consensus.cc:1261] T 4603c0d1e9fa433d8e03f6a4dadad53a P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:01.325153 25883 consensus_queue.cc:1035] T 4603c0d1e9fa433d8e03f6a4dadad53a P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:01.330104 25863 mvcc.cc:205] Tried to move back new op lower bound from 7022889129250136064 to 7022889128335568896. Current Snapshot: MvccSnapshot[applied={T|T < 7022889129250136064}]
I20240501 14:09:01.331780 24898 raft_consensus.cc:1261] T 6b90a26d65724b7e9be211b84b5da7fa P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Refusing update from remote peer f68136de04c94e5cb4a5f99881f02ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:01.332373 25999 consensus_queue.cc:1035] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:01.333441 25163 raft_consensus.cc:1261] T 6b90a26d65724b7e9be211b84b5da7fa P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Refusing update from remote peer f68136de04c94e5cb4a5f99881f02ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:01.333879 25861 consensus_queue.cc:1035] T 6b90a26d65724b7e9be211b84b5da7fa P f68136de04c94e5cb4a5f99881f02ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:01.334162 25030 raft_consensus.cc:1261] T e889bdce41804ba0a096a66c34c7c8f1 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Refusing update from remote peer 319fbe37b2ce4434a8d2fe879ca920a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:01.334645 25979 consensus_queue.cc:1035] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:01.355263 24501 raft_consensus.cc:1261] T e889bdce41804ba0a096a66c34c7c8f1 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Refusing update from remote peer 319fbe37b2ce4434a8d2fe879ca920a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:01.356326 25979 consensus_queue.cc:1035] T e889bdce41804ba0a096a66c34c7c8f1 P 319fbe37b2ce4434a8d2fe879ca920a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:01.370407 25911 mvcc.cc:205] Tried to move back new op lower bound from 7022889129250136064 to 7022889128335568896. Current Snapshot: MvccSnapshot[applied={T|T < 7022889129250136064}]
INSERT report
rows total: 2000
time total: 111.492 ms
time per row: 0.0557459 ms
I20240501 14:09:01.395907 24359 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43068:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:01.401608 24745 tablet_service.cc:1467] Processing CreateTablet for tablet c54aac554aa543a68f5aa0266ee2d63d (DEFAULT_TABLE table=rebalance_test_table_10 [id=aba2e96b6a9a4ef6b407247c26df0da7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.401696 25009 tablet_service.cc:1467] Processing CreateTablet for tablet c54aac554aa543a68f5aa0266ee2d63d (DEFAULT_TABLE table=rebalance_test_table_10 [id=aba2e96b6a9a4ef6b407247c26df0da7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.401948 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54aac554aa543a68f5aa0266ee2d63d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.401950 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54aac554aa543a68f5aa0266ee2d63d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.402552 24878 tablet_service.cc:1467] Processing CreateTablet for tablet c663405ba6d949868f0d31282fbecb09 (DEFAULT_TABLE table=rebalance_test_table_10 [id=aba2e96b6a9a4ef6b407247c26df0da7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.402818 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c663405ba6d949868f0d31282fbecb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.404058 25921 tablet_bootstrap.cc:492] T c54aac554aa543a68f5aa0266ee2d63d P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:09:01.404080 25922 tablet_bootstrap.cc:492] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:09:01.404690 24613 tablet_service.cc:1467] Processing CreateTablet for tablet c663405ba6d949868f0d31282fbecb09 (DEFAULT_TABLE table=rebalance_test_table_10 [id=aba2e96b6a9a4ef6b407247c26df0da7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.404729 24482 tablet_service.cc:1467] Processing CreateTablet for tablet c54aac554aa543a68f5aa0266ee2d63d (DEFAULT_TABLE table=rebalance_test_table_10 [id=aba2e96b6a9a4ef6b407247c26df0da7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.404963 25921 tablet_bootstrap.cc:654] T c54aac554aa543a68f5aa0266ee2d63d P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.404971 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c663405ba6d949868f0d31282fbecb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.404985 25922 tablet_bootstrap.cc:654] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.405272 25925 tablet_bootstrap.cc:492] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:09:01.406044 25925 tablet_bootstrap.cc:654] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.406466 25009 tablet_service.cc:1467] Processing CreateTablet for tablet bae65e02b4514cdeb733faeb8442821f (DEFAULT_TABLE table=rebalance_test_table_10 [id=aba2e96b6a9a4ef6b407247c26df0da7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.406477 24745 tablet_service.cc:1467] Processing CreateTablet for tablet bae65e02b4514cdeb733faeb8442821f (DEFAULT_TABLE table=rebalance_test_table_10 [id=aba2e96b6a9a4ef6b407247c26df0da7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.406697 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae65e02b4514cdeb733faeb8442821f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.406698 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae65e02b4514cdeb733faeb8442821f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.406808 25142 tablet_service.cc:1467] Processing CreateTablet for tablet c663405ba6d949868f0d31282fbecb09 (DEFAULT_TABLE table=rebalance_test_table_10 [id=aba2e96b6a9a4ef6b407247c26df0da7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.407048 25923 tablet_bootstrap.cc:492] T c663405ba6d949868f0d31282fbecb09 P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:09:01.407059 25142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c663405ba6d949868f0d31282fbecb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.407812 25923 tablet_bootstrap.cc:654] T c663405ba6d949868f0d31282fbecb09 P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.409056 25924 tablet_bootstrap.cc:492] T c663405ba6d949868f0d31282fbecb09 P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:09:01.409296 24482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54aac554aa543a68f5aa0266ee2d63d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.409611 25922 tablet_bootstrap.cc:492] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:09:01.409641 25923 tablet_bootstrap.cc:492] T c663405ba6d949868f0d31282fbecb09 P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:09:01.409686 25922 ts_tablet_manager.cc:1367] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:01.409703 25923 ts_tablet_manager.cc:1367] T c663405ba6d949868f0d31282fbecb09 P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:01.409878 25924 tablet_bootstrap.cc:654] T c663405ba6d949868f0d31282fbecb09 P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.410117 25922 raft_consensus.cc:348] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:01.410116 25923 raft_consensus.cc:348] T c663405ba6d949868f0d31282fbecb09 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.410218 25922 raft_consensus.cc:374] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.410218 25923 raft_consensus.cc:374] T c663405ba6d949868f0d31282fbecb09 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.410249 25922 raft_consensus.cc:729] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.410261 25923 raft_consensus.cc:729] T c663405ba6d949868f0d31282fbecb09 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.410372 25923 consensus_queue.cc:260] T c663405ba6d949868f0d31282fbecb09 P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.410372 25922 consensus_queue.cc:260] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:01.410631 25922 ts_tablet_manager.cc:1398] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:01.410631 25923 ts_tablet_manager.cc:1398] T c663405ba6d949868f0d31282fbecb09 P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:01.410797 25922 tablet_bootstrap.cc:492] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:09:01.411442 25920 tablet_bootstrap.cc:492] T c54aac554aa543a68f5aa0266ee2d63d P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:09:01.411466 25922 tablet_bootstrap.cc:654] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.412302 25920 tablet_bootstrap.cc:654] T c54aac554aa543a68f5aa0266ee2d63d P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.412590 25924 tablet_bootstrap.cc:492] T c663405ba6d949868f0d31282fbecb09 P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:09:01.412671 25924 ts_tablet_manager.cc:1367] T c663405ba6d949868f0d31282fbecb09 P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:01.412772 25922 tablet_bootstrap.cc:492] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:09:01.412914 25922 ts_tablet_manager.cc:1367] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:01.405084 24481 tablet_service.cc:1467] Processing CreateTablet for tablet bae65e02b4514cdeb733faeb8442821f (DEFAULT_TABLE table=rebalance_test_table_10 [id=aba2e96b6a9a4ef6b407247c26df0da7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.413163 24481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae65e02b4514cdeb733faeb8442821f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.413161 25924 raft_consensus.cc:348] T c663405ba6d949868f0d31282fbecb09 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.413251 25924 raft_consensus.cc:374] T c663405ba6d949868f0d31282fbecb09 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.413271 25924 raft_consensus.cc:729] T c663405ba6d949868f0d31282fbecb09 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.413332 25922 raft_consensus.cc:348] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:01.413374 25924 consensus_queue.cc:260] T c663405ba6d949868f0d31282fbecb09 P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.413420 25922 raft_consensus.cc:374] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.413453 25922 raft_consensus.cc:729] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.413558 25922 consensus_queue.cc:260] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:01.413619 25924 ts_tablet_manager.cc:1398] T c663405ba6d949868f0d31282fbecb09 P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:01.413807 25922 ts_tablet_manager.cc:1398] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:01.416606 25925 tablet_bootstrap.cc:492] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:09:01.416708 25925 ts_tablet_manager.cc:1367] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:01.416776 25921 tablet_bootstrap.cc:492] T c54aac554aa543a68f5aa0266ee2d63d P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:09:01.416887 25921 ts_tablet_manager.cc:1367] T c54aac554aa543a68f5aa0266ee2d63d P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240501 14:09:01.417217 25925 raft_consensus.cc:348] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.417323 25925 raft_consensus.cc:374] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.417356 25925 raft_consensus.cc:729] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.417352 25921 raft_consensus.cc:348] T c54aac554aa543a68f5aa0266ee2d63d P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:01.417451 25921 raft_consensus.cc:374] T c54aac554aa543a68f5aa0266ee2d63d P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.417479 25921 raft_consensus.cc:729] T c54aac554aa543a68f5aa0266ee2d63d P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.417475 25925 consensus_queue.cc:260] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.417594 25921 consensus_queue.cc:260] T c54aac554aa543a68f5aa0266ee2d63d P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:01.417636 25920 tablet_bootstrap.cc:492] T c54aac554aa543a68f5aa0266ee2d63d P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:09:01.417713 25920 ts_tablet_manager.cc:1367] T c54aac554aa543a68f5aa0266ee2d63d P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 14:09:01.417757 25925 ts_tablet_manager.cc:1398] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:01.417860 25921 ts_tablet_manager.cc:1398] T c54aac554aa543a68f5aa0266ee2d63d P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:01.418031 25921 tablet_bootstrap.cc:492] T bae65e02b4514cdeb733faeb8442821f P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:09:01.418177 25920 raft_consensus.cc:348] T c54aac554aa543a68f5aa0266ee2d63d P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:01.418270 25920 raft_consensus.cc:374] T c54aac554aa543a68f5aa0266ee2d63d P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.418298 25920 raft_consensus.cc:729] T c54aac554aa543a68f5aa0266ee2d63d P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.418419 25920 consensus_queue.cc:260] T c54aac554aa543a68f5aa0266ee2d63d P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:01.418677 25920 ts_tablet_manager.cc:1398] T c54aac554aa543a68f5aa0266ee2d63d P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:01.418833 25920 tablet_bootstrap.cc:492] T bae65e02b4514cdeb733faeb8442821f P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:09:01.418962 25921 tablet_bootstrap.cc:654] T bae65e02b4514cdeb733faeb8442821f P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.419647 25920 tablet_bootstrap.cc:654] T bae65e02b4514cdeb733faeb8442821f P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.419976 25921 tablet_bootstrap.cc:492] T bae65e02b4514cdeb733faeb8442821f P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:09:01.420034 25921 ts_tablet_manager.cc:1367] T bae65e02b4514cdeb733faeb8442821f P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:01.420445 25921 raft_consensus.cc:348] T bae65e02b4514cdeb733faeb8442821f P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:01.420538 25921 raft_consensus.cc:374] T bae65e02b4514cdeb733faeb8442821f P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.420568 25921 raft_consensus.cc:729] T bae65e02b4514cdeb733faeb8442821f P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.420676 25921 consensus_queue.cc:260] T bae65e02b4514cdeb733faeb8442821f P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:01.421047 25921 ts_tablet_manager.cc:1398] T bae65e02b4514cdeb733faeb8442821f P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:01.422179 25920 tablet_bootstrap.cc:492] T bae65e02b4514cdeb733faeb8442821f P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:09:01.422281 25920 ts_tablet_manager.cc:1367] T bae65e02b4514cdeb733faeb8442821f P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:09:01.422704 25920 raft_consensus.cc:348] T bae65e02b4514cdeb733faeb8442821f P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:01.422796 25920 raft_consensus.cc:374] T bae65e02b4514cdeb733faeb8442821f P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.422816 25920 raft_consensus.cc:729] T bae65e02b4514cdeb733faeb8442821f P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.422920 25920 consensus_queue.cc:260] T bae65e02b4514cdeb733faeb8442821f P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:01.423718 25920 ts_tablet_manager.cc:1398] T bae65e02b4514cdeb733faeb8442821f P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:09:01.533341 25917 raft_consensus.cc:482] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:01.533504 25917 raft_consensus.cc:504] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.533965 25917 leader_election.cc:288] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345), 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343)
I20240501 14:09:01.534348 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c663405ba6d949868f0d31282fbecb09" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3" is_pre_election: true
I20240501 14:09:01.534365 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c663405ba6d949868f0d31282fbecb09" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865" is_pre_election: true
I20240501 14:09:01.534505 25163 raft_consensus.cc:2454] T c663405ba6d949868f0d31282fbecb09 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 0.
I20240501 14:09:01.534513 24634 raft_consensus.cc:2454] T c663405ba6d949868f0d31282fbecb09 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 0.
I20240501 14:09:01.534781 24833 leader_election.cc:302] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, c2691129fd82431bb867cb1307e009b6; no voters:
I20240501 14:09:01.534966 25917 raft_consensus.cc:2781] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:01.535022 25917 raft_consensus.cc:482] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:01.535048 25917 raft_consensus.cc:3037] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.535867 25917 raft_consensus.cc:504] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.536217 25917 leader_election.cc:288] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 election: Requested vote from peers f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345), 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343)
I20240501 14:09:01.536595 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c663405ba6d949868f0d31282fbecb09" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865"
I20240501 14:09:01.536656 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c663405ba6d949868f0d31282fbecb09" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3"
I20240501 14:09:01.536727 25163 raft_consensus.cc:3037] T c663405ba6d949868f0d31282fbecb09 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.536757 24634 raft_consensus.cc:3037] T c663405ba6d949868f0d31282fbecb09 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.537618 24634 raft_consensus.cc:2454] T c663405ba6d949868f0d31282fbecb09 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 1.
I20240501 14:09:01.537668 25163 raft_consensus.cc:2454] T c663405ba6d949868f0d31282fbecb09 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 1.
I20240501 14:09:01.537948 24833 leader_election.cc:302] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, c2691129fd82431bb867cb1307e009b6; no voters:
I20240501 14:09:01.538210 25917 raft_consensus.cc:2781] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:01.538383 25917 raft_consensus.cc:686] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [term 1 LEADER]: Becoming Leader. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Running, Role: LEADER
I20240501 14:09:01.538569 25917 consensus_queue.cc:237] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.540318 24359 catalog_manager.cc:5484] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2691129fd82431bb867cb1307e009b6 (127.23.191.68). New cstate: current_term: 1 leader_uuid: "c2691129fd82431bb867cb1307e009b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:01.542194 25589 raft_consensus.cc:482] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:01.542296 25589 raft_consensus.cc:504] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:01.542707 25589 leader_election.cc:288] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:09:01.543030 24766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54aac554aa543a68f5aa0266ee2d63d" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" is_pre_election: true
I20240501 14:09:01.543084 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54aac554aa543a68f5aa0266ee2d63d" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:09:01.543182 24766 raft_consensus.cc:2454] T c54aac554aa543a68f5aa0266ee2d63d P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:09:01.543234 24501 raft_consensus.cc:2454] T c54aac554aa543a68f5aa0266ee2d63d P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:09:01.543455 24963 leader_election.cc:302] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:09:01.543613 25589 raft_consensus.cc:2781] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:01.543661 25589 raft_consensus.cc:482] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:01.543697 25589 raft_consensus.cc:3037] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.544659 25589 raft_consensus.cc:504] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:01.544989 25589 leader_election.cc:288] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Requested vote from peers 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311), fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561)
I20240501 14:09:01.545274 24766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54aac554aa543a68f5aa0266ee2d63d" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
I20240501 14:09:01.545332 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54aac554aa543a68f5aa0266ee2d63d" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:09:01.545379 24766 raft_consensus.cc:3037] T c54aac554aa543a68f5aa0266ee2d63d P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.545446 24501 raft_consensus.cc:3037] T c54aac554aa543a68f5aa0266ee2d63d P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.546152 24501 raft_consensus.cc:2454] T c54aac554aa543a68f5aa0266ee2d63d P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:09:01.546172 24766 raft_consensus.cc:2454] T c54aac554aa543a68f5aa0266ee2d63d P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:09:01.546415 24964 leader_election.cc:302] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e375044d63649c097759c4b667eff79, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:09:01.546579 25589 raft_consensus.cc:2781] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:01.546658 25589 raft_consensus.cc:686] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [term 1 LEADER]: Becoming Leader. State: Replica: 7e375044d63649c097759c4b667eff79, State: Running, Role: LEADER
I20240501 14:09:01.546787 25589 consensus_queue.cc:237] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } }
I20240501 14:09:01.549607 24359 catalog_manager.cc:5484] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e375044d63649c097759c4b667eff79 (127.23.191.69). New cstate: current_term: 1 leader_uuid: "7e375044d63649c097759c4b667eff79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:01.596635 26054 raft_consensus.cc:482] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:01.596769 26054 raft_consensus.cc:504] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:01.597205 26054 leader_election.cc:288] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:09:01.597661 24766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae65e02b4514cdeb733faeb8442821f" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" is_pre_election: true
I20240501 14:09:01.597708 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae65e02b4514cdeb733faeb8442821f" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" is_pre_election: true
I20240501 14:09:01.597798 24766 raft_consensus.cc:2454] T bae65e02b4514cdeb733faeb8442821f P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:09:01.597862 24501 raft_consensus.cc:2454] T bae65e02b4514cdeb733faeb8442821f P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 0.
I20240501 14:09:01.598157 24964 leader_election.cc:302] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e375044d63649c097759c4b667eff79, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:09:01.598421 26054 raft_consensus.cc:2781] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:01.598470 26054 raft_consensus.cc:482] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:01.598495 26054 raft_consensus.cc:3037] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.599671 26054 raft_consensus.cc:504] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:01.600019 26054 leader_election.cc:288] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Requested vote from peers fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65:43561), 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:09:01.600350 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae65e02b4514cdeb733faeb8442821f" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8ec7cff3a147e5adfc4eb329a23895"
I20240501 14:09:01.600430 24766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae65e02b4514cdeb733faeb8442821f" candidate_uuid: "7e375044d63649c097759c4b667eff79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
I20240501 14:09:01.600515 24501 raft_consensus.cc:3037] T bae65e02b4514cdeb733faeb8442821f P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.600530 24766 raft_consensus.cc:3037] T bae65e02b4514cdeb733faeb8442821f P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.601476 24766 raft_consensus.cc:2454] T bae65e02b4514cdeb733faeb8442821f P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:09:01.601476 24501 raft_consensus.cc:2454] T bae65e02b4514cdeb733faeb8442821f P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e375044d63649c097759c4b667eff79 in term 1.
I20240501 14:09:01.601785 24963 leader_election.cc:302] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319fbe37b2ce4434a8d2fe879ca920a7, 7e375044d63649c097759c4b667eff79; no voters:
I20240501 14:09:01.601943 26054 raft_consensus.cc:2781] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:01.602031 26054 raft_consensus.cc:686] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [term 1 LEADER]: Becoming Leader. State: Replica: 7e375044d63649c097759c4b667eff79, State: Running, Role: LEADER
I20240501 14:09:01.602173 26054 consensus_queue.cc:237] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } }
I20240501 14:09:01.604010 24359 catalog_manager.cc:5484] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e375044d63649c097759c4b667eff79 (127.23.191.69). New cstate: current_term: 1 leader_uuid: "7e375044d63649c097759c4b667eff79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:01.757506 26057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:01.757762 26057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:01.760999 26057 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 14:09:01.868911 24766 raft_consensus.cc:1261] T c54aac554aa543a68f5aa0266ee2d63d P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Refusing update from remote peer 7e375044d63649c097759c4b667eff79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:01.869379 25163 raft_consensus.cc:1261] T c663405ba6d949868f0d31282fbecb09 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Refusing update from remote peer c2691129fd82431bb867cb1307e009b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:01.869444 24634 raft_consensus.cc:1261] T c663405ba6d949868f0d31282fbecb09 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Refusing update from remote peer c2691129fd82431bb867cb1307e009b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:01.869951 24501 raft_consensus.cc:1261] T c54aac554aa543a68f5aa0266ee2d63d P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Refusing update from remote peer 7e375044d63649c097759c4b667eff79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:01.870002 25589 consensus_queue.cc:1035] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:01.870050 25917 consensus_queue.cc:1035] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:01.870630 25917 consensus_queue.cc:1035] T c663405ba6d949868f0d31282fbecb09 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:01.871448 26054 consensus_queue.cc:1035] T c54aac554aa543a68f5aa0266ee2d63d P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:01.873567 24501 raft_consensus.cc:1261] T bae65e02b4514cdeb733faeb8442821f P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Refusing update from remote peer 7e375044d63649c097759c4b667eff79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:01.873947 24766 raft_consensus.cc:1261] T bae65e02b4514cdeb733faeb8442821f P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Refusing update from remote peer 7e375044d63649c097759c4b667eff79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:01.874524 26054 consensus_queue.cc:1035] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:01.876183 25589 consensus_queue.cc:1035] T bae65e02b4514cdeb733faeb8442821f P 7e375044d63649c097759c4b667eff79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.433 ms
time per row: 0.0597166 ms
I20240501 14:09:01.928761 24356 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43068:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:01.936165 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 066db6793b254e6a98b7696973c07f94 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dcbb913647584bf786b1724819b06138]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.936501 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 066db6793b254e6a98b7696973c07f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.936730 24613 tablet_service.cc:1467] Processing CreateTablet for tablet 30ad6fcb42e341d88535668712e5fc64 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dcbb913647584bf786b1724819b06138]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.937040 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ad6fcb42e341d88535668712e5fc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.938796 26112 tablet_bootstrap.cc:492] T 066db6793b254e6a98b7696973c07f94 P 319fbe37b2ce4434a8d2fe879ca920a7: Bootstrap starting.
I20240501 14:09:01.939287 24614 tablet_service.cc:1467] Processing CreateTablet for tablet a887ae7493144f079a893af91fdb70f7 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dcbb913647584bf786b1724819b06138]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.939504 24614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a887ae7493144f079a893af91fdb70f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.939697 26112 tablet_bootstrap.cc:654] T 066db6793b254e6a98b7696973c07f94 P 319fbe37b2ce4434a8d2fe879ca920a7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.940738 26112 tablet_bootstrap.cc:492] T 066db6793b254e6a98b7696973c07f94 P 319fbe37b2ce4434a8d2fe879ca920a7: No bootstrap required, opened a new log
I20240501 14:09:01.941128 25009 tablet_service.cc:1467] Processing CreateTablet for tablet 066db6793b254e6a98b7696973c07f94 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dcbb913647584bf786b1724819b06138]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.940816 26112 ts_tablet_manager.cc:1367] T 066db6793b254e6a98b7696973c07f94 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:01.941395 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 066db6793b254e6a98b7696973c07f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.941649 26112 raft_consensus.cc:348] T 066db6793b254e6a98b7696973c07f94 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:01.941759 26112 raft_consensus.cc:374] T 066db6793b254e6a98b7696973c07f94 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.941787 26112 raft_consensus.cc:729] T 066db6793b254e6a98b7696973c07f94 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319fbe37b2ce4434a8d2fe879ca920a7, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.941906 26112 consensus_queue.cc:260] T 066db6793b254e6a98b7696973c07f94 P 319fbe37b2ce4434a8d2fe879ca920a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:01.942160 26112 ts_tablet_manager.cc:1398] T 066db6793b254e6a98b7696973c07f94 P 319fbe37b2ce4434a8d2fe879ca920a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:01.943398 26115 tablet_bootstrap.cc:492] T 066db6793b254e6a98b7696973c07f94 P 7e375044d63649c097759c4b667eff79: Bootstrap starting.
I20240501 14:09:01.943881 25143 tablet_service.cc:1467] Processing CreateTablet for tablet a887ae7493144f079a893af91fdb70f7 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dcbb913647584bf786b1724819b06138]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.943946 25142 tablet_service.cc:1467] Processing CreateTablet for tablet 30ad6fcb42e341d88535668712e5fc64 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dcbb913647584bf786b1724819b06138]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.944136 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a887ae7493144f079a893af91fdb70f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.944345 24878 tablet_service.cc:1467] Processing CreateTablet for tablet 30ad6fcb42e341d88535668712e5fc64 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dcbb913647584bf786b1724819b06138]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.944586 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ad6fcb42e341d88535668712e5fc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.944818 26115 tablet_bootstrap.cc:654] T 066db6793b254e6a98b7696973c07f94 P 7e375044d63649c097759c4b667eff79: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.945951 26115 tablet_bootstrap.cc:492] T 066db6793b254e6a98b7696973c07f94 P 7e375044d63649c097759c4b667eff79: No bootstrap required, opened a new log
I20240501 14:09:01.946028 26115 ts_tablet_manager.cc:1367] T 066db6793b254e6a98b7696973c07f94 P 7e375044d63649c097759c4b667eff79: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:01.946501 26115 raft_consensus.cc:348] T 066db6793b254e6a98b7696973c07f94 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:01.946539 24481 tablet_service.cc:1467] Processing CreateTablet for tablet 066db6793b254e6a98b7696973c07f94 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dcbb913647584bf786b1724819b06138]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.946602 26115 raft_consensus.cc:374] T 066db6793b254e6a98b7696973c07f94 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.946631 26115 raft_consensus.cc:729] T 066db6793b254e6a98b7696973c07f94 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e375044d63649c097759c4b667eff79, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.946733 26115 consensus_queue.cc:260] T 066db6793b254e6a98b7696973c07f94 P 7e375044d63649c097759c4b667eff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:01.946797 24481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 066db6793b254e6a98b7696973c07f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.947010 26115 ts_tablet_manager.cc:1398] T 066db6793b254e6a98b7696973c07f94 P 7e375044d63649c097759c4b667eff79: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:01.947556 26118 tablet_bootstrap.cc:492] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:09:01.944317 24877 tablet_service.cc:1467] Processing CreateTablet for tablet a887ae7493144f079a893af91fdb70f7 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dcbb913647584bf786b1724819b06138]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.947723 25142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ad6fcb42e341d88535668712e5fc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.947815 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a887ae7493144f079a893af91fdb70f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.948130 26113 tablet_bootstrap.cc:492] T 30ad6fcb42e341d88535668712e5fc64 P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:09:01.949225 26113 tablet_bootstrap.cc:654] T 30ad6fcb42e341d88535668712e5fc64 P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.950595 26118 tablet_bootstrap.cc:654] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.951664 26118 tablet_bootstrap.cc:492] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:09:01.951751 26118 ts_tablet_manager.cc:1367] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:01.952207 26118 raft_consensus.cc:348] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.952330 26118 raft_consensus.cc:374] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.952373 26118 raft_consensus.cc:729] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.952507 26118 consensus_queue.cc:260] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.952751 26118 ts_tablet_manager.cc:1398] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:01.952939 26118 tablet_bootstrap.cc:492] T a887ae7493144f079a893af91fdb70f7 P c2691129fd82431bb867cb1307e009b6: Bootstrap starting.
I20240501 14:09:01.953902 26118 tablet_bootstrap.cc:654] T a887ae7493144f079a893af91fdb70f7 P c2691129fd82431bb867cb1307e009b6: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.955916 26118 tablet_bootstrap.cc:492] T a887ae7493144f079a893af91fdb70f7 P c2691129fd82431bb867cb1307e009b6: No bootstrap required, opened a new log
I20240501 14:09:01.955996 26118 ts_tablet_manager.cc:1367] T a887ae7493144f079a893af91fdb70f7 P c2691129fd82431bb867cb1307e009b6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:01.956387 26118 raft_consensus.cc:348] T a887ae7493144f079a893af91fdb70f7 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:09:01.956543 26118 raft_consensus.cc:374] T a887ae7493144f079a893af91fdb70f7 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.956593 26118 raft_consensus.cc:729] T a887ae7493144f079a893af91fdb70f7 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.956737 26118 consensus_queue.cc:260] T a887ae7493144f079a893af91fdb70f7 P c2691129fd82431bb867cb1307e009b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:09:01.957082 26118 ts_tablet_manager.cc:1398] T a887ae7493144f079a893af91fdb70f7 P c2691129fd82431bb867cb1307e009b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:01.957233 26119 tablet_bootstrap.cc:492] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895: Bootstrap starting.
I20240501 14:09:01.958060 26117 tablet_bootstrap.cc:492] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:09:01.958928 26117 tablet_bootstrap.cc:654] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.959903 26119 tablet_bootstrap.cc:654] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.960019 26117 tablet_bootstrap.cc:492] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:09:01.960127 26117 ts_tablet_manager.cc:1367] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:01.960583 26117 raft_consensus.cc:348] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:09:01.960702 26117 raft_consensus.cc:374] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.960738 26117 raft_consensus.cc:729] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.960970 26113 tablet_bootstrap.cc:492] T 30ad6fcb42e341d88535668712e5fc64 P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:09:01.960944 26117 consensus_queue.cc:260] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:09:01.961068 26113 ts_tablet_manager.cc:1367] T 30ad6fcb42e341d88535668712e5fc64 P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:01.961244 26117 ts_tablet_manager.cc:1398] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:01.961421 26117 tablet_bootstrap.cc:492] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865: Bootstrap starting.
I20240501 14:09:01.961463 26113 raft_consensus.cc:348] T 30ad6fcb42e341d88535668712e5fc64 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.961575 26113 raft_consensus.cc:374] T 30ad6fcb42e341d88535668712e5fc64 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.961611 26113 raft_consensus.cc:729] T 30ad6fcb42e341d88535668712e5fc64 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.961724 26113 consensus_queue.cc:260] T 30ad6fcb42e341d88535668712e5fc64 P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.961987 26113 ts_tablet_manager.cc:1398] T 30ad6fcb42e341d88535668712e5fc64 P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:01.962234 26117 tablet_bootstrap.cc:654] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.962352 26113 tablet_bootstrap.cc:492] T a887ae7493144f079a893af91fdb70f7 P f68136de04c94e5cb4a5f99881f02ab3: Bootstrap starting.
I20240501 14:09:01.963187 26119 tablet_bootstrap.cc:492] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895: No bootstrap required, opened a new log
I20240501 14:09:01.963227 26113 tablet_bootstrap.cc:654] T a887ae7493144f079a893af91fdb70f7 P f68136de04c94e5cb4a5f99881f02ab3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.963760 26119 ts_tablet_manager.cc:1367] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:01.963976 26117 tablet_bootstrap.cc:492] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865: No bootstrap required, opened a new log
I20240501 14:09:01.964053 26117 ts_tablet_manager.cc:1367] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:01.964268 26113 tablet_bootstrap.cc:492] T a887ae7493144f079a893af91fdb70f7 P f68136de04c94e5cb4a5f99881f02ab3: No bootstrap required, opened a new log
I20240501 14:09:01.964339 26113 ts_tablet_manager.cc:1367] T a887ae7493144f079a893af91fdb70f7 P f68136de04c94e5cb4a5f99881f02ab3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:01.964352 26119 raft_consensus.cc:348] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:01.964462 26119 raft_consensus.cc:374] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.964427 26117 raft_consensus.cc:348] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.964491 26119 raft_consensus.cc:729] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.964507 26117 raft_consensus.cc:374] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.964536 26117 raft_consensus.cc:729] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.964613 26119 consensus_queue.cc:260] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:01.964627 26117 consensus_queue.cc:260] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:01.964720 26113 raft_consensus.cc:348] T a887ae7493144f079a893af91fdb70f7 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:09:01.964812 26113 raft_consensus.cc:374] T a887ae7493144f079a893af91fdb70f7 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.964866 26113 raft_consensus.cc:729] T a887ae7493144f079a893af91fdb70f7 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68136de04c94e5cb4a5f99881f02ab3, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.964893 26117 ts_tablet_manager.cc:1398] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:01.964964 26119 ts_tablet_manager.cc:1398] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:01.964956 26113 consensus_queue.cc:260] T a887ae7493144f079a893af91fdb70f7 P f68136de04c94e5cb4a5f99881f02ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:09:01.965200 26113 ts_tablet_manager.cc:1398] T a887ae7493144f079a893af91fdb70f7 P f68136de04c94e5cb4a5f99881f02ab3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:01.992285 25866 raft_consensus.cc:482] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:01.992419 25866 raft_consensus.cc:504] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:01.992862 25866 leader_election.cc:288] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635), 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:09:01.993289 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066db6793b254e6a98b7696973c07f94" candidate_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79" is_pre_election: true
I20240501 14:09:01.993358 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066db6793b254e6a98b7696973c07f94" candidate_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" is_pre_election: true
I20240501 14:09:01.993434 25030 raft_consensus.cc:2454] T 066db6793b254e6a98b7696973c07f94 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd8ec7cff3a147e5adfc4eb329a23895 in term 0.
I20240501 14:09:01.993526 24764 raft_consensus.cc:2454] T 066db6793b254e6a98b7696973c07f94 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd8ec7cff3a147e5adfc4eb329a23895 in term 0.
I20240501 14:09:01.993762 24435 leader_election.cc:302] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e375044d63649c097759c4b667eff79, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:09:01.993947 25866 raft_consensus.cc:2781] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:01.994014 25866 raft_consensus.cc:482] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:01.994058 25866 raft_consensus.cc:3037] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.994818 25866 raft_consensus.cc:504] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:01.995150 25866 leader_election.cc:288] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [CANDIDATE]: Term 1 election: Requested vote from peers 7e375044d63649c097759c4b667eff79 (127.23.191.69:45635), 319fbe37b2ce4434a8d2fe879ca920a7 (127.23.191.67:40311)
I20240501 14:09:01.995517 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066db6793b254e6a98b7696973c07f94" candidate_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319fbe37b2ce4434a8d2fe879ca920a7"
I20240501 14:09:01.995517 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066db6793b254e6a98b7696973c07f94" candidate_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e375044d63649c097759c4b667eff79"
I20240501 14:09:01.995628 24764 raft_consensus.cc:3037] T 066db6793b254e6a98b7696973c07f94 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.995628 25030 raft_consensus.cc:3037] T 066db6793b254e6a98b7696973c07f94 P 7e375044d63649c097759c4b667eff79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.996569 24764 raft_consensus.cc:2454] T 066db6793b254e6a98b7696973c07f94 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd8ec7cff3a147e5adfc4eb329a23895 in term 1.
I20240501 14:09:01.996569 25030 raft_consensus.cc:2454] T 066db6793b254e6a98b7696973c07f94 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd8ec7cff3a147e5adfc4eb329a23895 in term 1.
I20240501 14:09:01.996902 24435 leader_election.cc:302] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e375044d63649c097759c4b667eff79, fd8ec7cff3a147e5adfc4eb329a23895; no voters:
I20240501 14:09:01.997087 25866 raft_consensus.cc:2781] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:01.997241 25866 raft_consensus.cc:686] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [term 1 LEADER]: Becoming Leader. State: Replica: fd8ec7cff3a147e5adfc4eb329a23895, State: Running, Role: LEADER
I20240501 14:09:01.997396 25866 consensus_queue.cc:237] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } }
I20240501 14:09:01.999229 24355 catalog_manager.cc:5484] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd8ec7cff3a147e5adfc4eb329a23895 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd8ec7cff3a147e5adfc4eb329a23895" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 43561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:02.004750 25263 raft_consensus.cc:482] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:02.004887 25263 raft_consensus.cc:504] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:09:02.005263 25263 leader_election.cc:288] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:09:02.005718 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a887ae7493144f079a893af91fdb70f7" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3" is_pre_election: true
I20240501 14:09:02.005714 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a887ae7493144f079a893af91fdb70f7" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6" is_pre_election: true
I20240501 14:09:02.005869 24634 raft_consensus.cc:2454] T a887ae7493144f079a893af91fdb70f7 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:09:02.005872 24897 raft_consensus.cc:2454] T a887ae7493144f079a893af91fdb70f7 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:09:02.006196 25095 leader_election.cc:302] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:09:02.006470 25263 raft_consensus.cc:2781] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:02.006541 25263 raft_consensus.cc:482] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:02.006585 25263 raft_consensus.cc:3037] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:02.007344 25263 raft_consensus.cc:504] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:09:02.007640 25263 leader_election.cc:288] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Requested vote from peers c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971), f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345)
I20240501 14:09:02.007956 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a887ae7493144f079a893af91fdb70f7" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6"
I20240501 14:09:02.007987 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a887ae7493144f079a893af91fdb70f7" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3"
I20240501 14:09:02.008054 24897 raft_consensus.cc:3037] T a887ae7493144f079a893af91fdb70f7 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:02.008092 24634 raft_consensus.cc:3037] T a887ae7493144f079a893af91fdb70f7 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:02.008751 24897 raft_consensus.cc:2454] T a887ae7493144f079a893af91fdb70f7 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:09:02.008795 24634 raft_consensus.cc:2454] T a887ae7493144f079a893af91fdb70f7 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 1.
I20240501 14:09:02.009085 25098 leader_election.cc:302] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, c2691129fd82431bb867cb1307e009b6; no voters:
I20240501 14:09:02.009300 25263 raft_consensus.cc:2781] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:02.009372 25263 raft_consensus.cc:686] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [term 1 LEADER]: Becoming Leader. State: Replica: 786f81b7c57f452e9b463f721fc3e865, State: Running, Role: LEADER
I20240501 14:09:02.009487 25263 consensus_queue.cc:237] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } }
I20240501 14:09:02.011173 24355 catalog_manager.cc:5484] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70). New cstate: current_term: 1 leader_uuid: "786f81b7c57f452e9b463f721fc3e865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:02.044471 25263 raft_consensus.cc:482] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:02.044481 26052 raft_consensus.cc:482] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:02.044603 26052 raft_consensus.cc:504] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:02.044602 25263 raft_consensus.cc:504] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:02.045003 26052 leader_election.cc:288] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345), 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343)
I20240501 14:09:02.045022 25263 leader_election.cc:288] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345), c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971)
I20240501 14:09:02.045549 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad6fcb42e341d88535668712e5fc64" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6" is_pre_election: true
I20240501 14:09:02.045573 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad6fcb42e341d88535668712e5fc64" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865" is_pre_election: true
I20240501 14:09:02.045578 24633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad6fcb42e341d88535668712e5fc64" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3" is_pre_election: true
I20240501 14:09:02.045668 24897 raft_consensus.cc:2454] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:09:02.045668 25163 raft_consensus.cc:2454] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 0.
I20240501 14:09:02.045711 24633 raft_consensus.cc:2454] T 30ad6fcb42e341d88535668712e5fc64 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f81b7c57f452e9b463f721fc3e865 in term 0.
I20240501 14:09:02.045929 25098 leader_election.cc:302] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865, c2691129fd82431bb867cb1307e009b6; no voters:
I20240501 14:09:02.046020 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad6fcb42e341d88535668712e5fc64" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3" is_pre_election: true
I20240501 14:09:02.046085 25263 raft_consensus.cc:2781] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:02.046119 25263 raft_consensus.cc:482] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:02.046159 25263 raft_consensus.cc:3037] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:02.046144 24634 raft_consensus.cc:2454] T 30ad6fcb42e341d88535668712e5fc64 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 0.
I20240501 14:09:02.046406 24831 leader_election.cc:302] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2691129fd82431bb867cb1307e009b6, f68136de04c94e5cb4a5f99881f02ab3; no voters:
I20240501 14:09:02.046630 26052 raft_consensus.cc:2781] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:02.046679 26052 raft_consensus.cc:482] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:02.046756 26052 raft_consensus.cc:3037] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:02.047119 25263 raft_consensus.cc:504] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:02.047463 25263 leader_election.cc:288] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Requested vote from peers f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345), c2691129fd82431bb867cb1307e009b6 (127.23.191.68:44971)
I20240501 14:09:02.047667 26052 raft_consensus.cc:504] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:02.047842 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad6fcb42e341d88535668712e5fc64" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2691129fd82431bb867cb1307e009b6"
I20240501 14:09:02.048004 24897 raft_consensus.cc:2379] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 786f81b7c57f452e9b463f721fc3e865 in current term 1: Already voted for candidate c2691129fd82431bb867cb1307e009b6 in this term.
I20240501 14:09:02.048038 26052 leader_election.cc:288] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 election: Requested vote from peers f68136de04c94e5cb4a5f99881f02ab3 (127.23.191.66:40345), 786f81b7c57f452e9b463f721fc3e865 (127.23.191.70:35343)
I20240501 14:09:02.048470 24633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad6fcb42e341d88535668712e5fc64" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3"
I20240501 14:09:02.048503 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad6fcb42e341d88535668712e5fc64" candidate_uuid: "c2691129fd82431bb867cb1307e009b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786f81b7c57f452e9b463f721fc3e865"
I20240501 14:09:02.048573 24633 raft_consensus.cc:3037] T 30ad6fcb42e341d88535668712e5fc64 P f68136de04c94e5cb4a5f99881f02ab3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:02.048636 25163 raft_consensus.cc:2379] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c2691129fd82431bb867cb1307e009b6 in current term 1: Already voted for candidate 786f81b7c57f452e9b463f721fc3e865 in this term.
I20240501 14:09:02.049084 24634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad6fcb42e341d88535668712e5fc64" candidate_uuid: "786f81b7c57f452e9b463f721fc3e865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68136de04c94e5cb4a5f99881f02ab3"
I20240501 14:09:02.049432 25095 leader_election.cc:302] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 786f81b7c57f452e9b463f721fc3e865; no voters: c2691129fd82431bb867cb1307e009b6, f68136de04c94e5cb4a5f99881f02ab3
I20240501 14:09:02.049504 24633 raft_consensus.cc:2454] T 30ad6fcb42e341d88535668712e5fc64 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2691129fd82431bb867cb1307e009b6 in term 1.
I20240501 14:09:02.049690 25263 raft_consensus.cc:2726] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:09:02.049804 24831 leader_election.cc:302] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c2691129fd82431bb867cb1307e009b6, f68136de04c94e5cb4a5f99881f02ab3; no voters: 786f81b7c57f452e9b463f721fc3e865
I20240501 14:09:02.049989 26052 raft_consensus.cc:2781] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:02.050071 26052 raft_consensus.cc:686] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [term 1 LEADER]: Becoming Leader. State: Replica: c2691129fd82431bb867cb1307e009b6, State: Running, Role: LEADER
I20240501 14:09:02.050204 26052 consensus_queue.cc:237] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } }
I20240501 14:09:02.052099 24355 catalog_manager.cc:5484] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2691129fd82431bb867cb1307e009b6 (127.23.191.68). New cstate: current_term: 1 leader_uuid: "c2691129fd82431bb867cb1307e009b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:02.188261 26131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:02.188535 26131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:02.191679 26131 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240501 14:09:02.275055 25163 raft_consensus.cc:1261] T 30ad6fcb42e341d88535668712e5fc64 P 786f81b7c57f452e9b463f721fc3e865 [term 1 FOLLOWER]: Refusing update from remote peer c2691129fd82431bb867cb1307e009b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:02.275705 24633 raft_consensus.cc:1261] T 30ad6fcb42e341d88535668712e5fc64 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Refusing update from remote peer c2691129fd82431bb867cb1307e009b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:02.276269 25917 consensus_queue.cc:1035] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786f81b7c57f452e9b463f721fc3e865" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:02.276542 25917 consensus_queue.cc:1035] T 30ad6fcb42e341d88535668712e5fc64 P c2691129fd82431bb867cb1307e009b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:02.277796 24764 raft_consensus.cc:1261] T 066db6793b254e6a98b7696973c07f94 P 319fbe37b2ce4434a8d2fe879ca920a7 [term 1 FOLLOWER]: Refusing update from remote peer fd8ec7cff3a147e5adfc4eb329a23895: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:02.278887 24634 raft_consensus.cc:1261] T a887ae7493144f079a893af91fdb70f7 P f68136de04c94e5cb4a5f99881f02ab3 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:02.279109 25866 consensus_queue.cc:1035] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319fbe37b2ce4434a8d2fe879ca920a7" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 40311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:02.280022 25030 raft_consensus.cc:1261] T 066db6793b254e6a98b7696973c07f94 P 7e375044d63649c097759c4b667eff79 [term 1 FOLLOWER]: Refusing update from remote peer fd8ec7cff3a147e5adfc4eb329a23895: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:02.282080 24897 raft_consensus.cc:1261] T a887ae7493144f079a893af91fdb70f7 P c2691129fd82431bb867cb1307e009b6 [term 1 FOLLOWER]: Refusing update from remote peer 786f81b7c57f452e9b463f721fc3e865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:02.283684 25263 consensus_queue.cc:1035] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2691129fd82431bb867cb1307e009b6" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:02.286109 26127 consensus_queue.cc:1035] T 066db6793b254e6a98b7696973c07f94 P fd8ec7cff3a147e5adfc4eb329a23895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e375044d63649c097759c4b667eff79" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:02.288502 25883 consensus_queue.cc:1035] T a887ae7493144f079a893af91fdb70f7 P 786f81b7c57f452e9b463f721fc3e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68136de04c94e5cb4a5f99881f02ab3" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.927 ms
time per row: 0.0529636 ms
I20240501 14:09:02.557919 25142 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:02.559233 24745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:02.564960 24481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:02.569430 24614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:02.575048 25009 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:02.589124 24877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:02.705475 24745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:02.708391 25142 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:02.711324 24877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:02.712713 24614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:02.714619 25009 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:02.717034 24481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:02.853622 25142 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:02.854121 25009 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:02.860323 24877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:02.861274 24745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:02.869081 24614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:02.893550 24481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:02.992888 25142 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:02.995641 24877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:02.996250 24481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:02.996393 24745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:03.007205 24614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:03.020496 25009 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:03.127336 25142 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:03.129510 24745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:03.129756 24614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:03.133039 24877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:03.135612 25009 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:03.143034 24481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:03.272339 24481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:03.272490 24745 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:03.273607 25142 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:03.274228 24614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:03.276582 24877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:03.282778 25009 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
5f54f40816644fd3bef271c813eec707 | 127.23.191.126:46765 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.191.84:34733 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
319fbe37b2ce4434a8d2fe879ca920a7 | 127.23.191.67:40311 | HEALTHY | /B | 6 | 0
786f81b7c57f452e9b463f721fc3e865 | 127.23.191.70:35343 | HEALTHY | /C | 9 | 0
7e375044d63649c097759c4b667eff79 | 127.23.191.69:45635 | HEALTHY | /C | 9 | 0
c2691129fd82431bb867cb1307e009b6 | 127.23.191.68:44971 | HEALTHY | /B | 5 | 0
f68136de04c94e5cb4a5f99881f02ab3 | 127.23.191.66:40345 | HEALTHY | /A | 5 | 0
fd8ec7cff3a147e5adfc4eb329a23895 | 127.23.191.65:43561 | HEALTHY | /A | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/A | 2
/B | 2
/C | 2
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.23.191.65 | experimental | 127.23.191.65:43561
local_ip_for_outbound_sockets | 127.23.191.66 | experimental | 127.23.191.66:40345
local_ip_for_outbound_sockets | 127.23.191.67 | experimental | 127.23.191.67:40311
local_ip_for_outbound_sockets | 127.23.191.68 | experimental | 127.23.191.68:44971
local_ip_for_outbound_sockets | 127.23.191.69 | experimental | 127.23.191.69:45635
local_ip_for_outbound_sockets | 127.23.191.70 | experimental | 127.23.191.70:35343
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.191.65:43561
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.23.191.66:40345
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.191.67:40311
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.23.191.68:44971
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.23.191.69:45635
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.23.191.70:35343
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.191.84:34733 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_10 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_11 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 18
Median | 18
Third Quartile | 18
Maximum | 18
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 12
Tablets | 36
Replicas | 108
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:09:03.322175 24317 log_verifier.cc:126] Checking tablet 05b3283055ad40149a175eebfa58ee3c
I20240501 14:09:03.349453 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 05b3283055ad40149a175eebfa58ee3c
I20240501 14:09:03.349550 24317 log_verifier.cc:126] Checking tablet 066db6793b254e6a98b7696973c07f94
I20240501 14:09:03.374339 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 066db6793b254e6a98b7696973c07f94
I20240501 14:09:03.374404 24317 log_verifier.cc:126] Checking tablet 071272f7e339410890c23fb7e77465c1
I20240501 14:09:03.399230 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 071272f7e339410890c23fb7e77465c1
I20240501 14:09:03.399298 24317 log_verifier.cc:126] Checking tablet 1d557db356fa42afa39e059f0b56ee20
I20240501 14:09:03.423868 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1d557db356fa42afa39e059f0b56ee20
I20240501 14:09:03.423938 24317 log_verifier.cc:126] Checking tablet 29231eab3ff84873a05a65ad6513c4e5
I20240501 14:09:03.448323 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 29231eab3ff84873a05a65ad6513c4e5
I20240501 14:09:03.448388 24317 log_verifier.cc:126] Checking tablet 304de776104a4ab382e9274e3006be59
I20240501 14:09:03.472862 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 304de776104a4ab382e9274e3006be59
I20240501 14:09:03.472936 24317 log_verifier.cc:126] Checking tablet 30ad6fcb42e341d88535668712e5fc64
I20240501 14:09:03.497304 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 30ad6fcb42e341d88535668712e5fc64
I20240501 14:09:03.497375 24317 log_verifier.cc:126] Checking tablet 3973da211ea444b18da475e7e70c7e68
I20240501 14:09:03.521734 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3973da211ea444b18da475e7e70c7e68
I20240501 14:09:03.521806 24317 log_verifier.cc:126] Checking tablet 430ccd0aed3948f9a91560ace381d83c
I20240501 14:09:03.545990 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 430ccd0aed3948f9a91560ace381d83c
I20240501 14:09:03.546057 24317 log_verifier.cc:126] Checking tablet 44d3c8472d1543938d1bbbfd85b79447
I20240501 14:09:03.571676 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 44d3c8472d1543938d1bbbfd85b79447
I20240501 14:09:03.571753 24317 log_verifier.cc:126] Checking tablet 451c6589d9444bbfb539d82d65fa3be6
I20240501 14:09:03.597513 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 451c6589d9444bbfb539d82d65fa3be6
I20240501 14:09:03.597591 24317 log_verifier.cc:126] Checking tablet 4603c0d1e9fa433d8e03f6a4dadad53a
I20240501 14:09:03.622800 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4603c0d1e9fa433d8e03f6a4dadad53a
I20240501 14:09:03.622869 24317 log_verifier.cc:126] Checking tablet 5351479a8993455baa868e09bdd7530e
I20240501 14:09:03.648445 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5351479a8993455baa868e09bdd7530e
I20240501 14:09:03.648519 24317 log_verifier.cc:126] Checking tablet 5b6425c28a7a41fab2c271a06720e0c6
I20240501 14:09:03.674304 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5b6425c28a7a41fab2c271a06720e0c6
I20240501 14:09:03.674376 24317 log_verifier.cc:126] Checking tablet 6b90a26d65724b7e9be211b84b5da7fa
I20240501 14:09:03.698779 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6b90a26d65724b7e9be211b84b5da7fa
I20240501 14:09:03.698865 24317 log_verifier.cc:126] Checking tablet 78a1abcad3534f668299a245393d9022
I20240501 14:09:03.724871 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 78a1abcad3534f668299a245393d9022
I20240501 14:09:03.724938 24317 log_verifier.cc:126] Checking tablet 7b311d8b2b3f40f88c1580e9c24fcd4b
I20240501 14:09:03.749861 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7b311d8b2b3f40f88c1580e9c24fcd4b
I20240501 14:09:03.749924 24317 log_verifier.cc:126] Checking tablet 8023325ac74d43faa0817b49d3ef9707
I20240501 14:09:03.774145 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8023325ac74d43faa0817b49d3ef9707
I20240501 14:09:03.774210 24317 log_verifier.cc:126] Checking tablet 8060ab0e7d594d6d8dd1d789844062a7
I20240501 14:09:03.797834 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8060ab0e7d594d6d8dd1d789844062a7
I20240501 14:09:03.797907 24317 log_verifier.cc:126] Checking tablet 84643234d6424fab8eef66f4ae2eb254
I20240501 14:09:03.821256 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 84643234d6424fab8eef66f4ae2eb254
I20240501 14:09:03.821319 24317 log_verifier.cc:126] Checking tablet 8ef5de6cff3c4bc8856563f25222605f
I20240501 14:09:03.844709 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8ef5de6cff3c4bc8856563f25222605f
I20240501 14:09:03.844780 24317 log_verifier.cc:126] Checking tablet 90c5d177fbdc4be5b77fff955af6532b
I20240501 14:09:03.868520 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 90c5d177fbdc4be5b77fff955af6532b
I20240501 14:09:03.868584 24317 log_verifier.cc:126] Checking tablet 94b711edc7694340a0c4a68ef178d8fe
I20240501 14:09:03.892789 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 94b711edc7694340a0c4a68ef178d8fe
I20240501 14:09:03.892877 24317 log_verifier.cc:126] Checking tablet a5fd0fa373d74cd184b6626c559e3ee0
I20240501 14:09:03.917552 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a5fd0fa373d74cd184b6626c559e3ee0
I20240501 14:09:03.917618 24317 log_verifier.cc:126] Checking tablet a887ae7493144f079a893af91fdb70f7
I20240501 14:09:03.941514 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a887ae7493144f079a893af91fdb70f7
I20240501 14:09:03.941586 24317 log_verifier.cc:126] Checking tablet b8afe2fe38f84644a14abac37af3fed0
I20240501 14:09:03.965904 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b8afe2fe38f84644a14abac37af3fed0
I20240501 14:09:03.965967 24317 log_verifier.cc:126] Checking tablet bae65e02b4514cdeb733faeb8442821f
I20240501 14:09:03.989995 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bae65e02b4514cdeb733faeb8442821f
I20240501 14:09:03.990061 24317 log_verifier.cc:126] Checking tablet c4ae6674f0534b5eafc91d6579849ff0
I20240501 14:09:04.014017 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c4ae6674f0534b5eafc91d6579849ff0
I20240501 14:09:04.014088 24317 log_verifier.cc:126] Checking tablet c54aac554aa543a68f5aa0266ee2d63d
I20240501 14:09:04.040220 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c54aac554aa543a68f5aa0266ee2d63d
I20240501 14:09:04.040302 24317 log_verifier.cc:126] Checking tablet c663405ba6d949868f0d31282fbecb09
I20240501 14:09:04.064301 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c663405ba6d949868f0d31282fbecb09
I20240501 14:09:04.064375 24317 log_verifier.cc:126] Checking tablet d755aa2b9cd241ec8acede8f8fd46533
I20240501 14:09:04.088073 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d755aa2b9cd241ec8acede8f8fd46533
I20240501 14:09:04.088155 24317 log_verifier.cc:126] Checking tablet db3f5aae662b42b49a306f978985fd39
I20240501 14:09:04.111917 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet db3f5aae662b42b49a306f978985fd39
I20240501 14:09:04.111981 24317 log_verifier.cc:126] Checking tablet e2affc5e80214d0fae061d7b2bb4f013
I20240501 14:09:04.135174 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e2affc5e80214d0fae061d7b2bb4f013
I20240501 14:09:04.135246 24317 log_verifier.cc:126] Checking tablet e889bdce41804ba0a096a66c34c7c8f1
I20240501 14:09:04.159300 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e889bdce41804ba0a096a66c34c7c8f1
I20240501 14:09:04.159364 24317 log_verifier.cc:126] Checking tablet e9ff56af2d0040f4b3508807d1d4cb78
I20240501 14:09:04.183259 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e9ff56af2d0040f4b3508807d1d4cb78
I20240501 14:09:04.183332 24317 log_verifier.cc:126] Checking tablet ffb185b33baf4a0985355820d312f474
I20240501 14:09:04.207568 24317 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ffb185b33baf4a0985355820d312f474
I20240501 14:09:04.221349 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 24420
I20240501 14:09:04.229334 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 24551
I20240501 14:09:04.235504 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 24684
I20240501 14:09:04.240850 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 24815
I20240501 14:09:04.246098 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 24947
I20240501 14:09:04.251179 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 25079
I20240501 14:09:04.256553 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 24329
2024-05-01T14:09:04Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.Basic (14083 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (14083 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240501 14:09:04.350102 24317 test_util.cc:268] Using random seed: 124950416
I20240501 14:09:04.350852 24317 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:09:04.350903 24317 ts_itest-base.cc:113] --------------
I20240501 14:09:04.350914 24317 ts_itest-base.cc:114] 5 tablet servers
I20240501 14:09:04.350924 24317 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:09:04.350934 24317 ts_itest-base.cc:116] --------------
2024-05-01T14:09:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:09:04Z Disabled control of system clock
I20240501 14:09:04.358825 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.191.126:33133
--webserver_interface=127.23.191.126
--webserver_port=0
--builtin_ntp_servers=127.23.191.84:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.191.126:33133
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 14:09:04.464774 26464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:04.465049 26464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:04.465108 26464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:04.468151 26464 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:04.468228 26464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:09:04.468256 26464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:04.468288 26464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:09:04.468322 26464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:09:04.472045 26464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.23.191.126:33133
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.191.126:33133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.23.191.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.26464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:04.473014 26464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:04.474153 26464 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 14:09:04.482092 26464 server_base.cc:1034] running on GCE node
W20240501 14:09:04.482636 26472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:04.483063 26469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:04.483610 26470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:04.484150 26464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:04.485152 26464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:04.486306 26464 hybrid_clock.cc:648] HybridClock initialized: now 1714572544486299 us; error 34 us; skew 500 ppm
I20240501 14:09:04.488231 26464 webserver.cc:460] Webserver started at http://127.23.191.126:42591/ using document root <none> and password file <none>
I20240501 14:09:04.488782 26464 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:04.488870 26464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:04.489060 26464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:04.490741 26464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0787437adee24972a68291a1cf531c86"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-c4g3"
I20240501 14:09:04.491207 26464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0787437adee24972a68291a1cf531c86"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-c4g3"
I20240501 14:09:04.494562 26464 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:04.496821 26478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.497869 26464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:04.498003 26464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0787437adee24972a68291a1cf531c86"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-c4g3"
I20240501 14:09:04.498093 26464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:04.515110 26464 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:04.515755 26464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:04.515918 26464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:04.522238 26464 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.126:33133
I20240501 14:09:04.522243 26530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.126:33133 every 8 connection(s)
I20240501 14:09:04.523231 26464 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:09:04.523716 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 26464
I20240501 14:09:04.523823 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:09:04.525976 26531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:04.531055 26531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86: Bootstrap starting.
I20240501 14:09:04.533396 26531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:04.534077 26531 log.cc:824] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:04.535629 26531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86: No bootstrap required, opened a new log
I20240501 14:09:04.538142 26531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0787437adee24972a68291a1cf531c86" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 33133 } }
I20240501 14:09:04.538322 26531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:04.538376 26531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0787437adee24972a68291a1cf531c86, State: Initialized, Role: FOLLOWER
I20240501 14:09:04.538772 26531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0787437adee24972a68291a1cf531c86" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 33133 } }
I20240501 14:09:04.538880 26531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:04.538936 26531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:04.539042 26531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:04.539758 26531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0787437adee24972a68291a1cf531c86" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 33133 } }
I20240501 14:09:04.540069 26531 leader_election.cc:302] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0787437adee24972a68291a1cf531c86; no voters:
I20240501 14:09:04.540323 26531 leader_election.cc:288] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:04.540416 26536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:04.540609 26536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [term 1 LEADER]: Becoming Leader. State: Replica: 0787437adee24972a68291a1cf531c86, State: Running, Role: LEADER
I20240501 14:09:04.540962 26536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0787437adee24972a68291a1cf531c86" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 33133 } }
I20240501 14:09:04.541404 26531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:09:04.542615 26538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0787437adee24972a68291a1cf531c86. Latest consensus state: current_term: 1 leader_uuid: "0787437adee24972a68291a1cf531c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0787437adee24972a68291a1cf531c86" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 33133 } } }
I20240501 14:09:04.542655 26537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0787437adee24972a68291a1cf531c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0787437adee24972a68291a1cf531c86" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 33133 } } }
I20240501 14:09:04.542729 26538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:04.542752 26537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86 [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:04.543572 26542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:09:04.546505 26542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:09:04.551767 26542 catalog_manager.cc:1347] Generated new cluster ID: 3bd2eb91a26a476f94f6a0fa460a80fc
I20240501 14:09:04.551826 26542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:09:04.574119 26542 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:09:04.575023 26542 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:09:04.581218 26542 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0787437adee24972a68291a1cf531c86: Generated new TSK 0
I20240501 14:09:04.581888 26542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:09:04.590392 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.65:0
--local_ip_for_outbound_sockets=127.23.191.65
--webserver_interface=127.23.191.65
--webserver_port=0
--tserver_master_addrs=127.23.191.126:33133
--builtin_ntp_servers=127.23.191.84:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:04.696559 26555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:04.696768 26555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:04.696815 26555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:04.699855 26555 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:04.699926 26555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:04.700021 26555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.65
I20240501 14:09:04.703773 26555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.23.191.65
--webserver_port=0
--tserver_master_addrs=127.23.191.126:33133
--never_fsync=true
--heap_profile_path=/tmp/kudu.26555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.65
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:04.704672 26555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:04.705849 26555 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:04.714339 26561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:04.714845 26564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:04.715251 26562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:04.715992 26555 server_base.cc:1034] running on GCE node
I20240501 14:09:04.716343 26555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:04.716917 26555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:04.718067 26555 hybrid_clock.cc:648] HybridClock initialized: now 1714572544718053 us; error 31 us; skew 500 ppm
I20240501 14:09:04.719938 26555 webserver.cc:460] Webserver started at http://127.23.191.65:43015/ using document root <none> and password file <none>
I20240501 14:09:04.720476 26555 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:04.720532 26555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:04.720741 26555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:04.722462 26555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b30d93291d764bf6bb89bff916b0fd4d"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-c4g3"
I20240501 14:09:04.722931 26555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b30d93291d764bf6bb89bff916b0fd4d"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-c4g3"
I20240501 14:09:04.726336 26555 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:04.728567 26570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.729593 26555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:04.729702 26555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b30d93291d764bf6bb89bff916b0fd4d"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-c4g3"
I20240501 14:09:04.729800 26555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:04.746059 26555 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:04.746666 26555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:04.746834 26555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:04.747367 26555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:04.748270 26555 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:04.748325 26555 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.748378 26555 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:04.748401 26555 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.756414 26555 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.65:40481
I20240501 14:09:04.756443 26683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.65:40481 every 8 connection(s)
I20240501 14:09:04.757357 26555 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:09:04.765697 26684 heartbeater.cc:347] Connected to a master server at 127.23.191.126:33133
I20240501 14:09:04.765923 26684 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:04.766429 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 26555
I20240501 14:09:04.766556 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:09:04.766566 26684 heartbeater.cc:511] Master 127.23.191.126:33133 requested a full tablet report, sending...
I20240501 14:09:04.768131 26494 ts_manager.cc:194] Registered new tserver with Master: b30d93291d764bf6bb89bff916b0fd4d (127.23.191.65:40481)
I20240501 14:09:04.768689 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.66:0
--local_ip_for_outbound_sockets=127.23.191.66
--webserver_interface=127.23.191.66
--webserver_port=0
--tserver_master_addrs=127.23.191.126:33133
--builtin_ntp_servers=127.23.191.84:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:04.769712 26494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.65:47357
W20240501 14:09:04.873109 26687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:04.873332 26687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:04.873373 26687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:04.876418 26687 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:04.876495 26687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:04.876564 26687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.66
I20240501 14:09:04.880198 26687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.23.191.66
--webserver_port=0
--tserver_master_addrs=127.23.191.126:33133
--never_fsync=true
--heap_profile_path=/tmp/kudu.26687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.66
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:04.881134 26687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:04.882220 26687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:04.890470 26695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:04.890918 26692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:04.891371 26693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:04.892038 26687 server_base.cc:1034] running on GCE node
I20240501 14:09:04.892417 26687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:04.893033 26687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:04.894204 26687 hybrid_clock.cc:648] HybridClock initialized: now 1714572544894187 us; error 30 us; skew 500 ppm
I20240501 14:09:04.896338 26687 webserver.cc:460] Webserver started at http://127.23.191.66:41311/ using document root <none> and password file <none>
I20240501 14:09:04.897059 26687 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:04.897140 26687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:04.897342 26687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:04.899084 26687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8590de10980a4c9081ce2be1ec6ba085"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-c4g3"
I20240501 14:09:04.899555 26687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8590de10980a4c9081ce2be1ec6ba085"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-c4g3"
I20240501 14:09:04.903016 26687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:04.905184 26701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.906147 26687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:04.906255 26687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8590de10980a4c9081ce2be1ec6ba085"
format_stamp: "Formatted at 2024-05-01 14:09:04 on dist-test-slave-c4g3"
I20240501 14:09:04.906348 26687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:04.937896 26687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:04.938544 26687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:04.938706 26687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:04.939263 26687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:04.940158 26687 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:04.940208 26687 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.940260 26687 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:04.940291 26687 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:04.948487 26687 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.66:41925
I20240501 14:09:04.948518 26814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.66:41925 every 8 connection(s)
I20240501 14:09:04.949456 26687 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:09:04.955292 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 26687
I20240501 14:09:04.955394 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:09:04.957659 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.67:0
--local_ip_for_outbound_sockets=127.23.191.67
--webserver_interface=127.23.191.67
--webserver_port=0
--tserver_master_addrs=127.23.191.126:33133
--builtin_ntp_servers=127.23.191.84:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:04.958010 26815 heartbeater.cc:347] Connected to a master server at 127.23.191.126:33133
I20240501 14:09:04.958251 26815 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:04.958849 26815 heartbeater.cc:511] Master 127.23.191.126:33133 requested a full tablet report, sending...
I20240501 14:09:04.959908 26494 ts_manager.cc:194] Registered new tserver with Master: 8590de10980a4c9081ce2be1ec6ba085 (127.23.191.66:41925)
I20240501 14:09:04.960458 26494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.66:42341
W20240501 14:09:05.062489 26818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:05.062712 26818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:05.062758 26818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:05.065801 26818 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:05.065871 26818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:05.065953 26818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.67
I20240501 14:09:05.069530 26818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.23.191.67
--webserver_port=0
--tserver_master_addrs=127.23.191.126:33133
--never_fsync=true
--heap_profile_path=/tmp/kudu.26818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.67
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:05.070415 26818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:05.071493 26818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:05.079574 26823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:05.079988 26826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:05.080392 26824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:05.081068 26818 server_base.cc:1034] running on GCE node
I20240501 14:09:05.081490 26818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:05.082067 26818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:05.083226 26818 hybrid_clock.cc:648] HybridClock initialized: now 1714572545083205 us; error 34 us; skew 500 ppm
I20240501 14:09:05.085150 26818 webserver.cc:460] Webserver started at http://127.23.191.67:42359/ using document root <none> and password file <none>
I20240501 14:09:05.085691 26818 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:05.085752 26818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:05.085920 26818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:05.087580 26818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7740cb162a804761851773499c663171"
format_stamp: "Formatted at 2024-05-01 14:09:05 on dist-test-slave-c4g3"
I20240501 14:09:05.088042 26818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7740cb162a804761851773499c663171"
format_stamp: "Formatted at 2024-05-01 14:09:05 on dist-test-slave-c4g3"
I20240501 14:09:05.091338 26818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 14:09:05.093523 26832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:05.094504 26818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:09:05.094610 26818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7740cb162a804761851773499c663171"
format_stamp: "Formatted at 2024-05-01 14:09:05 on dist-test-slave-c4g3"
I20240501 14:09:05.094696 26818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:05.119617 26818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:05.120241 26818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:05.120421 26818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:05.120988 26818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:05.121907 26818 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:05.121958 26818 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:05.122012 26818 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:05.122036 26818 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:05.130169 26818 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.67:42637
I20240501 14:09:05.130193 26945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.67:42637 every 8 connection(s)
I20240501 14:09:05.131131 26818 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:09:05.133000 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 26818
I20240501 14:09:05.133112 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:09:05.135367 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.68:0
--local_ip_for_outbound_sockets=127.23.191.68
--webserver_interface=127.23.191.68
--webserver_port=0
--tserver_master_addrs=127.23.191.126:33133
--builtin_ntp_servers=127.23.191.84:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:05.140866 26946 heartbeater.cc:347] Connected to a master server at 127.23.191.126:33133
I20240501 14:09:05.141127 26946 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:05.141707 26946 heartbeater.cc:511] Master 127.23.191.126:33133 requested a full tablet report, sending...
I20240501 14:09:05.142840 26494 ts_manager.cc:194] Registered new tserver with Master: 7740cb162a804761851773499c663171 (127.23.191.67:42637)
I20240501 14:09:05.143610 26494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.67:34873
W20240501 14:09:05.242473 26949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:05.242713 26949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:05.242779 26949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:05.245884 26949 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:05.245952 26949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:05.246021 26949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.68
I20240501 14:09:05.249637 26949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.23.191.68
--webserver_port=0
--tserver_master_addrs=127.23.191.126:33133
--never_fsync=true
--heap_profile_path=/tmp/kudu.26949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.68
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:05.250550 26949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:05.251724 26949 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:05.259986 26957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:05.260437 26954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:05.260949 26955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:05.261438 26949 server_base.cc:1034] running on GCE node
I20240501 14:09:05.261828 26949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:05.262387 26949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:05.263545 26949 hybrid_clock.cc:648] HybridClock initialized: now 1714572545263524 us; error 35 us; skew 500 ppm
I20240501 14:09:05.265555 26949 webserver.cc:460] Webserver started at http://127.23.191.68:36025/ using document root <none> and password file <none>
I20240501 14:09:05.266096 26949 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:05.266157 26949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:05.266328 26949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:05.267963 26949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c8aa7bf697b44d5796504cffb8d9281f"
format_stamp: "Formatted at 2024-05-01 14:09:05 on dist-test-slave-c4g3"
I20240501 14:09:05.268425 26949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c8aa7bf697b44d5796504cffb8d9281f"
format_stamp: "Formatted at 2024-05-01 14:09:05 on dist-test-slave-c4g3"
I20240501 14:09:05.271703 26949 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:05.273869 26963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:05.274855 26949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:05.274963 26949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c8aa7bf697b44d5796504cffb8d9281f"
format_stamp: "Formatted at 2024-05-01 14:09:05 on dist-test-slave-c4g3"
I20240501 14:09:05.275048 26949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:05.298401 26949 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:05.299037 26949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:05.299192 26949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:05.299717 26949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:05.300590 26949 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:05.300640 26949 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:05.300693 26949 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:05.300721 26949 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:05.308915 26949 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.68:36633
I20240501 14:09:05.308934 27076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.68:36633 every 8 connection(s)
I20240501 14:09:05.310029 26949 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:09:05.311517 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 26949
I20240501 14:09:05.311640 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:09:05.314157 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.69:0
--local_ip_for_outbound_sockets=127.23.191.69
--webserver_interface=127.23.191.69
--webserver_port=0
--tserver_master_addrs=127.23.191.126:33133
--builtin_ntp_servers=127.23.191.84:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:05.319909 27077 heartbeater.cc:347] Connected to a master server at 127.23.191.126:33133
I20240501 14:09:05.320143 27077 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:05.320658 27077 heartbeater.cc:511] Master 127.23.191.126:33133 requested a full tablet report, sending...
I20240501 14:09:05.321743 26494 ts_manager.cc:194] Registered new tserver with Master: c8aa7bf697b44d5796504cffb8d9281f (127.23.191.68:36633)
I20240501 14:09:05.322329 26494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.68:33747
W20240501 14:09:05.420814 27080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:05.421057 27080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:05.421118 27080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:05.424139 27080 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:05.424209 27080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:05.424302 27080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.69
I20240501 14:09:05.427867 27080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.23.191.69
--webserver_port=0
--tserver_master_addrs=127.23.191.126:33133
--never_fsync=true
--heap_profile_path=/tmp/kudu.27080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.69
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:05.428743 27080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:05.429944 27080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:05.438377 27086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:05.438887 27085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:05.439743 27088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:05.440080 27080 server_base.cc:1034] running on GCE node
I20240501 14:09:05.440505 27080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:05.441143 27080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:05.442313 27080 hybrid_clock.cc:648] HybridClock initialized: now 1714572545442296 us; error 37 us; skew 500 ppm
I20240501 14:09:05.444612 27080 webserver.cc:460] Webserver started at http://127.23.191.69:44669/ using document root <none> and password file <none>
I20240501 14:09:05.445222 27080 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:05.445283 27080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:05.445457 27080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:05.447129 27080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f431b0cfba744c00be3281d8097508fc"
format_stamp: "Formatted at 2024-05-01 14:09:05 on dist-test-slave-c4g3"
I20240501 14:09:05.447602 27080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f431b0cfba744c00be3281d8097508fc"
format_stamp: "Formatted at 2024-05-01 14:09:05 on dist-test-slave-c4g3"
I20240501 14:09:05.451087 27080 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:05.453312 27094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:05.454334 27080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:05.454439 27080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f431b0cfba744c00be3281d8097508fc"
format_stamp: "Formatted at 2024-05-01 14:09:05 on dist-test-slave-c4g3"
I20240501 14:09:05.454530 27080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:05.469570 27080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:05.470176 27080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:05.470331 27080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:05.470916 27080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:05.471956 27080 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:05.472011 27080 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:05.472064 27080 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:05.472091 27080 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:05.481032 27080 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.69:39411
I20240501 14:09:05.481066 27207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.69:39411 every 8 connection(s)
I20240501 14:09:05.482081 27080 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:09:05.490161 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 27080
I20240501 14:09:05.490278 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:09:05.490799 27208 heartbeater.cc:347] Connected to a master server at 127.23.191.126:33133
I20240501 14:09:05.491034 27208 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:05.491631 27208 heartbeater.cc:511] Master 127.23.191.126:33133 requested a full tablet report, sending...
I20240501 14:09:05.492664 26494 ts_manager.cc:194] Registered new tserver with Master: f431b0cfba744c00be3281d8097508fc (127.23.191.69:39411)
I20240501 14:09:05.493244 26494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.69:60325
I20240501 14:09:05.504215 24317 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:09:05.512903 26494 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34118:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:09:05.530086 26749 tablet_service.cc:1467] Processing CreateTablet for tablet 3530c575bdf84807bf64f33c042f00d9 (DEFAULT_TABLE table=TestTable [id=615388447435428f8354b5b96f27629c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.531265 26749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3530c575bdf84807bf64f33c042f00d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.535813 27011 tablet_service.cc:1467] Processing CreateTablet for tablet 3530c575bdf84807bf64f33c042f00d9 (DEFAULT_TABLE table=TestTable [id=615388447435428f8354b5b96f27629c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.536793 27011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3530c575bdf84807bf64f33c042f00d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.542199 27225 tablet_bootstrap.cc:492] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f: Bootstrap starting.
I20240501 14:09:05.542658 27224 tablet_bootstrap.cc:492] T 3530c575bdf84807bf64f33c042f00d9 P 8590de10980a4c9081ce2be1ec6ba085: Bootstrap starting.
I20240501 14:09:05.544627 27224 tablet_bootstrap.cc:654] T 3530c575bdf84807bf64f33c042f00d9 P 8590de10980a4c9081ce2be1ec6ba085: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.544627 27225 tablet_bootstrap.cc:654] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.544893 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 3530c575bdf84807bf64f33c042f00d9 (DEFAULT_TABLE table=TestTable [id=615388447435428f8354b5b96f27629c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.545483 27225 log.cc:824] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:05.545483 27224 log.cc:824] T 3530c575bdf84807bf64f33c042f00d9 P 8590de10980a4c9081ce2be1ec6ba085: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:05.546006 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3530c575bdf84807bf64f33c042f00d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.547089 27225 tablet_bootstrap.cc:492] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f: No bootstrap required, opened a new log
I20240501 14:09:05.547261 27225 ts_tablet_manager.cc:1367] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 14:09:05.547561 27224 tablet_bootstrap.cc:492] T 3530c575bdf84807bf64f33c042f00d9 P 8590de10980a4c9081ce2be1ec6ba085: No bootstrap required, opened a new log
I20240501 14:09:05.547758 27224 ts_tablet_manager.cc:1367] T 3530c575bdf84807bf64f33c042f00d9 P 8590de10980a4c9081ce2be1ec6ba085: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240501 14:09:05.550208 27225 raft_consensus.cc:348] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8590de10980a4c9081ce2be1ec6ba085" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 41925 } } peers { permanent_uuid: "c8aa7bf697b44d5796504cffb8d9281f" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 36633 } } peers { permanent_uuid: "f431b0cfba744c00be3281d8097508fc" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39411 } }
I20240501 14:09:05.550391 27225 raft_consensus.cc:374] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.550442 27225 raft_consensus.cc:729] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8aa7bf697b44d5796504cffb8d9281f, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.550843 27225 consensus_queue.cc:260] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8590de10980a4c9081ce2be1ec6ba085" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 41925 } } peers { permanent_uuid: "c8aa7bf697b44d5796504cffb8d9281f" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 36633 } } peers { permanent_uuid: "f431b0cfba744c00be3281d8097508fc" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39411 } }
I20240501 14:09:05.551538 27077 heartbeater.cc:503] Master 127.23.191.126:33133 was elected leader, sending a full tablet report...
I20240501 14:09:05.551491 27224 raft_consensus.cc:348] T 3530c575bdf84807bf64f33c042f00d9 P 8590de10980a4c9081ce2be1ec6ba085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8590de10980a4c9081ce2be1ec6ba085" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 41925 } } peers { permanent_uuid: "c8aa7bf697b44d5796504cffb8d9281f" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 36633 } } peers { permanent_uuid: "f431b0cfba744c00be3281d8097508fc" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39411 } }
I20240501 14:09:05.551680 27225 ts_tablet_manager.cc:1398] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20240501 14:09:05.551715 27224 raft_consensus.cc:374] T 3530c575bdf84807bf64f33c042f00d9 P 8590de10980a4c9081ce2be1ec6ba085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.551777 27224 raft_consensus.cc:729] T 3530c575bdf84807bf64f33c042f00d9 P 8590de10980a4c9081ce2be1ec6ba085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8590de10980a4c9081ce2be1ec6ba085, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.552207 27224 consensus_queue.cc:260] T 3530c575bdf84807bf64f33c042f00d9 P 8590de10980a4c9081ce2be1ec6ba085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8590de10980a4c9081ce2be1ec6ba085" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 41925 } } peers { permanent_uuid: "c8aa7bf697b44d5796504cffb8d9281f" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 36633 } } peers { permanent_uuid: "f431b0cfba744c00be3281d8097508fc" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39411 } }
I20240501 14:09:05.553087 27224 ts_tablet_manager.cc:1398] T 3530c575bdf84807bf64f33c042f00d9 P 8590de10980a4c9081ce2be1ec6ba085: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20240501 14:09:05.553328 26815 heartbeater.cc:503] Master 127.23.191.126:33133 was elected leader, sending a full tablet report...
I20240501 14:09:05.554473 27230 tablet_bootstrap.cc:492] T 3530c575bdf84807bf64f33c042f00d9 P f431b0cfba744c00be3281d8097508fc: Bootstrap starting.
I20240501 14:09:05.556340 27230 tablet_bootstrap.cc:654] T 3530c575bdf84807bf64f33c042f00d9 P f431b0cfba744c00be3281d8097508fc: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.557066 27230 log.cc:824] T 3530c575bdf84807bf64f33c042f00d9 P f431b0cfba744c00be3281d8097508fc: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:05.558661 27230 tablet_bootstrap.cc:492] T 3530c575bdf84807bf64f33c042f00d9 P f431b0cfba744c00be3281d8097508fc: No bootstrap required, opened a new log
I20240501 14:09:05.558856 27230 ts_tablet_manager.cc:1367] T 3530c575bdf84807bf64f33c042f00d9 P f431b0cfba744c00be3281d8097508fc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
W20240501 14:09:05.561162 27078 tablet.cc:2369] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:05.562008 27230 raft_consensus.cc:348] T 3530c575bdf84807bf64f33c042f00d9 P f431b0cfba744c00be3281d8097508fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8590de10980a4c9081ce2be1ec6ba085" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 41925 } } peers { permanent_uuid: "c8aa7bf697b44d5796504cffb8d9281f" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 36633 } } peers { permanent_uuid: "f431b0cfba744c00be3281d8097508fc" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39411 } }
I20240501 14:09:05.562206 27230 raft_consensus.cc:374] T 3530c575bdf84807bf64f33c042f00d9 P f431b0cfba744c00be3281d8097508fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.562249 27230 raft_consensus.cc:729] T 3530c575bdf84807bf64f33c042f00d9 P f431b0cfba744c00be3281d8097508fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f431b0cfba744c00be3281d8097508fc, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.562606 27230 consensus_queue.cc:260] T 3530c575bdf84807bf64f33c042f00d9 P f431b0cfba744c00be3281d8097508fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8590de10980a4c9081ce2be1ec6ba085" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 41925 } } peers { permanent_uuid: "c8aa7bf697b44d5796504cffb8d9281f" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 36633 } } peers { permanent_uuid: "f431b0cfba744c00be3281d8097508fc" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39411 } }
I20240501 14:09:05.563273 27208 heartbeater.cc:503] Master 127.23.191.126:33133 was elected leader, sending a full tablet report...
I20240501 14:09:05.563419 27230 ts_tablet_manager.cc:1398] T 3530c575bdf84807bf64f33c042f00d9 P f431b0cfba744c00be3281d8097508fc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20240501 14:09:05.700659 26816 tablet.cc:2369] T 3530c575bdf84807bf64f33c042f00d9 P 8590de10980a4c9081ce2be1ec6ba085: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:05.729934 27228 raft_consensus.cc:482] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:05.730155 27228 raft_consensus.cc:504] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8590de10980a4c9081ce2be1ec6ba085" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 41925 } } peers { permanent_uuid: "c8aa7bf697b44d5796504cffb8d9281f" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 36633 } } peers { permanent_uuid: "f431b0cfba744c00be3281d8097508fc" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39411 } }
I20240501 14:09:05.731315 27228 leader_election.cc:288] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8590de10980a4c9081ce2be1ec6ba085 (127.23.191.66:41925), f431b0cfba744c00be3281d8097508fc (127.23.191.69:39411)
W20240501 14:09:05.732982 27209 tablet.cc:2369] T 3530c575bdf84807bf64f33c042f00d9 P f431b0cfba744c00be3281d8097508fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:05.735653 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3530c575bdf84807bf64f33c042f00d9" candidate_uuid: "c8aa7bf697b44d5796504cffb8d9281f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f431b0cfba744c00be3281d8097508fc" is_pre_election: true
I20240501 14:09:05.735955 26769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3530c575bdf84807bf64f33c042f00d9" candidate_uuid: "c8aa7bf697b44d5796504cffb8d9281f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8590de10980a4c9081ce2be1ec6ba085" is_pre_election: true
I20240501 14:09:05.735987 27162 raft_consensus.cc:2454] T 3530c575bdf84807bf64f33c042f00d9 P f431b0cfba744c00be3281d8097508fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8aa7bf697b44d5796504cffb8d9281f in term 0.
I20240501 14:09:05.736227 26769 raft_consensus.cc:2454] T 3530c575bdf84807bf64f33c042f00d9 P 8590de10980a4c9081ce2be1ec6ba085 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8aa7bf697b44d5796504cffb8d9281f in term 0.
I20240501 14:09:05.736474 26964 leader_election.cc:302] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8aa7bf697b44d5796504cffb8d9281f, f431b0cfba744c00be3281d8097508fc; no voters:
I20240501 14:09:05.736752 27228 raft_consensus.cc:2781] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:05.736851 27228 raft_consensus.cc:482] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:05.736901 27228 raft_consensus.cc:3037] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.737726 27228 raft_consensus.cc:504] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8590de10980a4c9081ce2be1ec6ba085" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 41925 } } peers { permanent_uuid: "c8aa7bf697b44d5796504cffb8d9281f" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 36633 } } peers { permanent_uuid: "f431b0cfba744c00be3281d8097508fc" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39411 } }
I20240501 14:09:05.738083 27228 leader_election.cc:288] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [CANDIDATE]: Term 1 election: Requested vote from peers 8590de10980a4c9081ce2be1ec6ba085 (127.23.191.66:41925), f431b0cfba744c00be3281d8097508fc (127.23.191.69:39411)
I20240501 14:09:05.738415 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3530c575bdf84807bf64f33c042f00d9" candidate_uuid: "c8aa7bf697b44d5796504cffb8d9281f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f431b0cfba744c00be3281d8097508fc"
I20240501 14:09:05.738533 27162 raft_consensus.cc:3037] T 3530c575bdf84807bf64f33c042f00d9 P f431b0cfba744c00be3281d8097508fc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.738507 26769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3530c575bdf84807bf64f33c042f00d9" candidate_uuid: "c8aa7bf697b44d5796504cffb8d9281f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8590de10980a4c9081ce2be1ec6ba085"
I20240501 14:09:05.738664 26769 raft_consensus.cc:3037] T 3530c575bdf84807bf64f33c042f00d9 P 8590de10980a4c9081ce2be1ec6ba085 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.772269 26684 heartbeater.cc:503] Master 127.23.191.126:33133 was elected leader, sending a full tablet report...
I20240501 14:09:05.834982 27162 raft_consensus.cc:2454] T 3530c575bdf84807bf64f33c042f00d9 P f431b0cfba744c00be3281d8097508fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8aa7bf697b44d5796504cffb8d9281f in term 1.
I20240501 14:09:05.834990 26769 raft_consensus.cc:2454] T 3530c575bdf84807bf64f33c042f00d9 P 8590de10980a4c9081ce2be1ec6ba085 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8aa7bf697b44d5796504cffb8d9281f in term 1.
I20240501 14:09:05.835518 26965 leader_election.cc:302] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8590de10980a4c9081ce2be1ec6ba085, c8aa7bf697b44d5796504cffb8d9281f; no voters:
I20240501 14:09:05.835803 27228 raft_consensus.cc:2781] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.835994 27228 raft_consensus.cc:686] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [term 1 LEADER]: Becoming Leader. State: Replica: c8aa7bf697b44d5796504cffb8d9281f, State: Running, Role: LEADER
I20240501 14:09:05.836359 27228 consensus_queue.cc:237] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8590de10980a4c9081ce2be1ec6ba085" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 41925 } } peers { permanent_uuid: "c8aa7bf697b44d5796504cffb8d9281f" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 36633 } } peers { permanent_uuid: "f431b0cfba744c00be3281d8097508fc" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39411 } }
I20240501 14:09:05.839779 26495 catalog_manager.cc:5484] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f reported cstate change: term changed from 0 to 1, leader changed from <none> to c8aa7bf697b44d5796504cffb8d9281f (127.23.191.68). New cstate: current_term: 1 leader_uuid: "c8aa7bf697b44d5796504cffb8d9281f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8590de10980a4c9081ce2be1ec6ba085" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 41925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8aa7bf697b44d5796504cffb8d9281f" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 36633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f431b0cfba744c00be3281d8097508fc" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39411 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:05.887799 24317 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:09:05.889331 24317 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b30d93291d764bf6bb89bff916b0fd4d to finish bootstrapping
I20240501 14:09:05.895090 24317 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8590de10980a4c9081ce2be1ec6ba085 to finish bootstrapping
I20240501 14:09:05.899963 24317 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7740cb162a804761851773499c663171 to finish bootstrapping
I20240501 14:09:05.905629 24317 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c8aa7bf697b44d5796504cffb8d9281f to finish bootstrapping
I20240501 14:09:05.910717 24317 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f431b0cfba744c00be3281d8097508fc to finish bootstrapping
I20240501 14:09:06.100257 26749 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.100268 26880 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.100564 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.100991 27011 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.102535 26618 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.145687 26946 heartbeater.cc:503] Master 127.23.191.126:33133 was elected leader, sending a full tablet report...
I20240501 14:09:06.191119 26880 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.192018 26749 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.192559 26618 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.194344 27011 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.195129 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.321491 27228 consensus_queue.cc:1035] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8590de10980a4c9081ce2be1ec6ba085" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 41925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:06.327311 27228 consensus_queue.cc:1035] T 3530c575bdf84807bf64f33c042f00d9 P c8aa7bf697b44d5796504cffb8d9281f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f431b0cfba744c00be3281d8097508fc" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:06.412690 26880 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.415385 26749 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.416503 26618 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.424991 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.425022 27011 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.504189 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.504994 26618 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.505292 26749 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.505973 26880 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.506484 27011 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.596210 26880 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.596525 26618 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.597440 26749 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.598605 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.605312 27011 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.634619 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 26555
I20240501 14:09:06.639351 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 26687
I20240501 14:09:06.643982 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 26818
I20240501 14:09:06.648236 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 26949
I20240501 14:09:06.652722 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 27080
I20240501 14:09:06.657462 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 26464
2024-05-01T14:09:06Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (2327 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2327 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20240501 14:09:06.677745 24317 test_util.cc:268] Using random seed: 127278060
I20240501 14:09:06.678493 24317 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:09:06.678534 24317 ts_itest-base.cc:113] --------------
I20240501 14:09:06.678548 24317 ts_itest-base.cc:114] 7 tablet servers
I20240501 14:09:06.678560 24317 ts_itest-base.cc:115] 2 replicas per TS
I20240501 14:09:06.678575 24317 ts_itest-base.cc:116] --------------
2024-05-01T14:09:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:09:06Z Disabled control of system clock
I20240501 14:09:06.685793 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.191.126:38485
--webserver_interface=127.23.191.126
--webserver_port=0
--builtin_ntp_servers=127.23.191.84:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.191.126:38485
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 14:09:06.788304 27418 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 14:09:06.788527 27418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:06.788553 27418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:06.788589 27418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:06.791532 27418 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:06.791595 27418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:09:06.791618 27418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:06.791641 27418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:09:06.791657 27418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:09:06.795212 27418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.23.191.126:38485
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.191.126:38485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.23.191.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.27418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:06.796020 27418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:06.797061 27418 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:06.804013 27423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:06.804311 27424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:06.804890 27426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:06.805397 27418 server_base.cc:1034] running on GCE node
I20240501 14:09:06.805827 27418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:06.806696 27418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:06.807840 27418 hybrid_clock.cc:648] HybridClock initialized: now 1714572546807827 us; error 25 us; skew 500 ppm
I20240501 14:09:06.809623 27418 webserver.cc:460] Webserver started at http://127.23.191.126:37621/ using document root <none> and password file <none>
I20240501 14:09:06.810132 27418 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:06.810192 27418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:06.810376 27418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:06.811930 27418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "25f42fba200742378c49fb096885390d"
format_stamp: "Formatted at 2024-05-01 14:09:06 on dist-test-slave-c4g3"
I20240501 14:09:06.812373 27418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "25f42fba200742378c49fb096885390d"
format_stamp: "Formatted at 2024-05-01 14:09:06 on dist-test-slave-c4g3"
I20240501 14:09:06.815438 27418 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:06.817503 27432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:06.818423 27418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:06.818533 27418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
uuid: "25f42fba200742378c49fb096885390d"
format_stamp: "Formatted at 2024-05-01 14:09:06 on dist-test-slave-c4g3"
I20240501 14:09:06.818609 27418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:06.843925 27418 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:06.844592 27418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:06.844754 27418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:06.850895 27418 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.126:38485
I20240501 14:09:06.850904 27484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.126:38485 every 8 connection(s)
I20240501 14:09:06.851815 27418 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:09:06.854272 27485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:06.859011 27485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d: Bootstrap starting.
I20240501 14:09:06.860106 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 27418
I20240501 14:09:06.860211 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:09:06.861267 27485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:06.861999 27485 log.cc:824] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:06.863601 27485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d: No bootstrap required, opened a new log
I20240501 14:09:06.866065 27485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f42fba200742378c49fb096885390d" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 38485 } }
I20240501 14:09:06.866240 27485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:06.866300 27485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f42fba200742378c49fb096885390d, State: Initialized, Role: FOLLOWER
I20240501 14:09:06.866691 27485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f42fba200742378c49fb096885390d" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 38485 } }
I20240501 14:09:06.866842 27485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:06.866910 27485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:06.867025 27485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:06.867867 27485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f42fba200742378c49fb096885390d" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 38485 } }
I20240501 14:09:06.868183 27485 leader_election.cc:302] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25f42fba200742378c49fb096885390d; no voters:
I20240501 14:09:06.868430 27485 leader_election.cc:288] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:06.868772 27490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:06.869032 27490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [term 1 LEADER]: Becoming Leader. State: Replica: 25f42fba200742378c49fb096885390d, State: Running, Role: LEADER
I20240501 14:09:06.869313 27490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f42fba200742378c49fb096885390d" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 38485 } }
I20240501 14:09:06.869495 27485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:09:06.870702 27491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25f42fba200742378c49fb096885390d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f42fba200742378c49fb096885390d" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 38485 } } }
I20240501 14:09:06.870731 27492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25f42fba200742378c49fb096885390d. Latest consensus state: current_term: 1 leader_uuid: "25f42fba200742378c49fb096885390d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f42fba200742378c49fb096885390d" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 38485 } } }
I20240501 14:09:06.870819 27491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:06.870827 27492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:06.871881 27500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:09:06.874346 27500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:09:06.879091 27500 catalog_manager.cc:1347] Generated new cluster ID: 23b80d62d634496a861f167aa7f65470
I20240501 14:09:06.879145 27500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:09:06.892452 27500 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:09:06.893584 27500 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:09:06.901795 27500 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 25f42fba200742378c49fb096885390d: Generated new TSK 0
I20240501 14:09:06.902571 27500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:09:06.905648 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.65:0
--local_ip_for_outbound_sockets=127.23.191.65
--webserver_interface=127.23.191.65
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--builtin_ntp_servers=127.23.191.84:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:07.008208 27509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:07.008430 27509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:07.008491 27509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:07.011512 27509 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:07.011575 27509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:07.011657 27509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.65
I20240501 14:09:07.015188 27509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.23.191.65
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--never_fsync=true
--heap_profile_path=/tmp/kudu.27509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.65
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:07.016009 27509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:07.017118 27509 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:07.025228 27515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:07.025655 27517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:07.026240 27514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:07.026765 27509 server_base.cc:1034] running on GCE node
I20240501 14:09:07.027144 27509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:07.027719 27509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:07.028882 27509 hybrid_clock.cc:648] HybridClock initialized: now 1714572547028870 us; error 26 us; skew 500 ppm
I20240501 14:09:07.030501 27509 webserver.cc:460] Webserver started at http://127.23.191.65:42245/ using document root <none> and password file <none>
I20240501 14:09:07.031035 27509 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:07.031086 27509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:07.031244 27509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:07.032819 27509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c9cc3f1efaf44a4895d3109beabe5312"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.033265 27509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c9cc3f1efaf44a4895d3109beabe5312"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.036218 27509 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240501 14:09:07.038261 27523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.039193 27509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:07.039286 27509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c9cc3f1efaf44a4895d3109beabe5312"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.039369 27509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:07.060547 27509 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:07.061269 27509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:07.061424 27509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:07.061957 27509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:07.062829 27509 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:07.062875 27509 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.062932 27509 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:07.062961 27509 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.070782 27509 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.65:34669
I20240501 14:09:07.070827 27636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.65:34669 every 8 connection(s)
I20240501 14:09:07.071732 27509 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:09:07.079708 27637 heartbeater.cc:347] Connected to a master server at 127.23.191.126:38485
I20240501 14:09:07.079931 27637 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:07.080437 27637 heartbeater.cc:511] Master 127.23.191.126:38485 requested a full tablet report, sending...
I20240501 14:09:07.081426 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 27509
I20240501 14:09:07.081537 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:09:07.081967 27449 ts_manager.cc:194] Registered new tserver with Master: c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:07.083297 27449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.65:33239
I20240501 14:09:07.083771 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.66:0
--local_ip_for_outbound_sockets=127.23.191.66
--webserver_interface=127.23.191.66
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--builtin_ntp_servers=127.23.191.84:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:07.188778 27640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:07.189047 27640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:07.189091 27640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:07.192180 27640 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:07.192251 27640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:07.192317 27640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.66
I20240501 14:09:07.196049 27640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.23.191.66
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--never_fsync=true
--heap_profile_path=/tmp/kudu.27640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.66
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:07.196951 27640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:07.198040 27640 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:07.206461 27646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:07.206739 27645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:07.207432 27648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:07.207875 27640 server_base.cc:1034] running on GCE node
I20240501 14:09:07.208281 27640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:07.208889 27640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:07.210041 27640 hybrid_clock.cc:648] HybridClock initialized: now 1714572547210022 us; error 40 us; skew 500 ppm
I20240501 14:09:07.211889 27640 webserver.cc:460] Webserver started at http://127.23.191.66:39585/ using document root <none> and password file <none>
I20240501 14:09:07.212435 27640 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:07.212494 27640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:07.212668 27640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:07.214388 27640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "fec89cd1e3ba491eab76811aae4af170"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.214879 27640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "fec89cd1e3ba491eab76811aae4af170"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.218207 27640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:07.220419 27654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.221500 27640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:07.221618 27640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fec89cd1e3ba491eab76811aae4af170"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.221707 27640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:07.237684 27640 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:07.238351 27640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:07.238517 27640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:07.239077 27640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:07.239995 27640 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:07.240043 27640 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.240098 27640 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:07.240123 27640 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.248541 27640 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.66:44265
I20240501 14:09:07.248575 27767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.66:44265 every 8 connection(s)
I20240501 14:09:07.249526 27640 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:09:07.257606 27768 heartbeater.cc:347] Connected to a master server at 127.23.191.126:38485
I20240501 14:09:07.257826 27768 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:07.258322 27768 heartbeater.cc:511] Master 127.23.191.126:38485 requested a full tablet report, sending...
I20240501 14:09:07.259275 27449 ts_manager.cc:194] Registered new tserver with Master: fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:07.259534 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 27640
I20240501 14:09:07.259632 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:09:07.259877 27449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.66:58849
I20240501 14:09:07.261473 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.67:0
--local_ip_for_outbound_sockets=127.23.191.67
--webserver_interface=127.23.191.67
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--builtin_ntp_servers=127.23.191.84:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:07.365870 27771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:07.366086 27771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:07.366129 27771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:07.369107 27771 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:07.369185 27771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:07.369253 27771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.67
I20240501 14:09:07.372931 27771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.23.191.67
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--never_fsync=true
--heap_profile_path=/tmp/kudu.27771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.67
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:07.373811 27771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:07.374876 27771 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:07.383119 27779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:07.383438 27776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:07.384119 27777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:07.384572 27771 server_base.cc:1034] running on GCE node
I20240501 14:09:07.384991 27771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:07.385538 27771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:07.386689 27771 hybrid_clock.cc:648] HybridClock initialized: now 1714572547386676 us; error 28 us; skew 500 ppm
I20240501 14:09:07.388461 27771 webserver.cc:460] Webserver started at http://127.23.191.67:37123/ using document root <none> and password file <none>
I20240501 14:09:07.389034 27771 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:07.389094 27771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:07.389273 27771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:07.390919 27771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d1f8294a92e74484ab4a653a4ecf84e5"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.391377 27771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d1f8294a92e74484ab4a653a4ecf84e5"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.394582 27771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:07.396661 27785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.397648 27771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:07.397751 27771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d1f8294a92e74484ab4a653a4ecf84e5"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.397838 27771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:07.420600 27771 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:07.421348 27771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:07.421511 27771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:07.422060 27771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:07.423015 27771 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:07.423082 27771 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.423138 27771 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:07.423163 27771 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.431299 27771 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.67:45647
I20240501 14:09:07.431340 27898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.67:45647 every 8 connection(s)
I20240501 14:09:07.432201 27771 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:09:07.436910 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 27771
I20240501 14:09:07.437003 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:09:07.439070 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.68:0
--local_ip_for_outbound_sockets=127.23.191.68
--webserver_interface=127.23.191.68
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--builtin_ntp_servers=127.23.191.84:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:07.440995 27899 heartbeater.cc:347] Connected to a master server at 127.23.191.126:38485
I20240501 14:09:07.441241 27899 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:07.441859 27899 heartbeater.cc:511] Master 127.23.191.126:38485 requested a full tablet report, sending...
I20240501 14:09:07.443014 27449 ts_manager.cc:194] Registered new tserver with Master: d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:07.443590 27449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.67:47467
W20240501 14:09:07.544349 27902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:07.544582 27902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:07.544646 27902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:07.547720 27902 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:07.547804 27902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:07.547894 27902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.68
I20240501 14:09:07.551668 27902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.23.191.68
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--never_fsync=true
--heap_profile_path=/tmp/kudu.27902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.68
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:07.552608 27902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:07.553791 27902 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:07.561834 27910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:07.562151 27907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:07.562769 27908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:07.563243 27902 server_base.cc:1034] running on GCE node
I20240501 14:09:07.563639 27902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:07.564199 27902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:07.565361 27902 hybrid_clock.cc:648] HybridClock initialized: now 1714572547565343 us; error 32 us; skew 500 ppm
I20240501 14:09:07.567217 27902 webserver.cc:460] Webserver started at http://127.23.191.68:37467/ using document root <none> and password file <none>
I20240501 14:09:07.567742 27902 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:07.567795 27902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:07.567953 27902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:07.569600 27902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4c37ce2bb5d844c09969e644d463cf17"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.570080 27902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4c37ce2bb5d844c09969e644d463cf17"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.573359 27902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:07.575506 27916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.576488 27902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:07.576586 27902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4c37ce2bb5d844c09969e644d463cf17"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.576682 27902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:07.590655 27902 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:07.591310 27902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:07.591467 27902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:07.592002 27902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:07.592922 27902 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:07.592969 27902 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.593012 27902 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:07.593048 27902 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.601388 27902 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.68:43375
I20240501 14:09:07.601430 28029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.68:43375 every 8 connection(s)
I20240501 14:09:07.602403 27902 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:09:07.604879 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 27902
I20240501 14:09:07.604990 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:09:07.607072 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.69:0
--local_ip_for_outbound_sockets=127.23.191.69
--webserver_interface=127.23.191.69
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--builtin_ntp_servers=127.23.191.84:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:07.611784 28030 heartbeater.cc:347] Connected to a master server at 127.23.191.126:38485
I20240501 14:09:07.612012 28030 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:07.612548 28030 heartbeater.cc:511] Master 127.23.191.126:38485 requested a full tablet report, sending...
I20240501 14:09:07.613687 27449 ts_manager.cc:194] Registered new tserver with Master: 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68:43375)
I20240501 14:09:07.614421 27449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.68:57713
W20240501 14:09:07.715236 28033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:07.715548 28033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:07.715617 28033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:07.720139 28033 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:07.720273 28033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:07.720373 28033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.69
I20240501 14:09:07.725863 28033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.23.191.69
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--never_fsync=true
--heap_profile_path=/tmp/kudu.28033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.69
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:07.727146 28033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:07.728657 28033 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:07.739365 28038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:07.739639 28039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:07.740316 28041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:07.740756 28033 server_base.cc:1034] running on GCE node
I20240501 14:09:07.741261 28033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:07.741852 28033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:07.742988 28033 hybrid_clock.cc:648] HybridClock initialized: now 1714572547742963 us; error 38 us; skew 500 ppm
I20240501 14:09:07.745030 28033 webserver.cc:460] Webserver started at http://127.23.191.69:39265/ using document root <none> and password file <none>
I20240501 14:09:07.745573 28033 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:07.745625 28033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:07.745788 28033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:07.747471 28033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ae4fd219bcae42e3969310879588dc32"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.747927 28033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ae4fd219bcae42e3969310879588dc32"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.751408 28033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:07.753844 28047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.754915 28033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:07.755023 28033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ae4fd219bcae42e3969310879588dc32"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.755124 28033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:07.765949 28033 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:07.766597 28033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:07.766757 28033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:07.767300 28033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:07.768245 28033 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:07.768304 28033 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.768347 28033 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:07.768374 28033 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.776818 28033 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.69:39239
I20240501 14:09:07.776885 28160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.69:39239 every 8 connection(s)
I20240501 14:09:07.777798 28033 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:09:07.783317 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 28033
I20240501 14:09:07.783430 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:09:07.785760 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.70:0
--local_ip_for_outbound_sockets=127.23.191.70
--webserver_interface=127.23.191.70
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--builtin_ntp_servers=127.23.191.84:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:07.787118 28161 heartbeater.cc:347] Connected to a master server at 127.23.191.126:38485
I20240501 14:09:07.787357 28161 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:07.787912 28161 heartbeater.cc:511] Master 127.23.191.126:38485 requested a full tablet report, sending...
I20240501 14:09:07.789011 27449 ts_manager.cc:194] Registered new tserver with Master: ae4fd219bcae42e3969310879588dc32 (127.23.191.69:39239)
I20240501 14:09:07.789732 27449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.69:59677
W20240501 14:09:07.895274 28164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:07.895505 28164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:07.895550 28164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:07.898603 28164 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:07.898669 28164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:07.898734 28164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.70
I20240501 14:09:07.902457 28164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.23.191.70
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--never_fsync=true
--heap_profile_path=/tmp/kudu.28164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.70
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:07.903349 28164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:07.904423 28164 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:07.913137 28169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:07.913486 28170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:07.914171 28172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:07.914657 28164 server_base.cc:1034] running on GCE node
I20240501 14:09:07.915017 28164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:07.915570 28164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:07.916720 28164 hybrid_clock.cc:648] HybridClock initialized: now 1714572547916696 us; error 38 us; skew 500 ppm
I20240501 14:09:07.918578 28164 webserver.cc:460] Webserver started at http://127.23.191.70:33593/ using document root <none> and password file <none>
I20240501 14:09:07.919106 28164 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:07.919162 28164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:07.919348 28164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:07.921008 28164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "3625e18e2ea14d43aa3b86b014fd89f5"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.921454 28164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "3625e18e2ea14d43aa3b86b014fd89f5"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.924633 28164 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:07.926775 28178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.927742 28164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:07.927848 28164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3625e18e2ea14d43aa3b86b014fd89f5"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:07.927932 28164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:07.937582 28164 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:07.938171 28164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:07.938338 28164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:07.938825 28164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:07.939659 28164 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:07.939707 28164 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.939756 28164 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:07.939783 28164 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.947921 28164 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.70:33907
I20240501 14:09:07.947952 28291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.70:33907 every 8 connection(s)
I20240501 14:09:07.948781 28164 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:09:07.952527 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 28164
I20240501 14:09:07.952638 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:09:07.954720 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.71:0
--local_ip_for_outbound_sockets=127.23.191.71
--webserver_interface=127.23.191.71
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--builtin_ntp_servers=127.23.191.84:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:07.958878 28292 heartbeater.cc:347] Connected to a master server at 127.23.191.126:38485
I20240501 14:09:07.959136 28292 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:07.959838 28292 heartbeater.cc:511] Master 127.23.191.126:38485 requested a full tablet report, sending...
I20240501 14:09:07.960951 27449 ts_manager.cc:194] Registered new tserver with Master: 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70:33907)
I20240501 14:09:07.961509 27449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.70:58877
W20240501 14:09:08.061010 28295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:08.061264 28295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:08.061326 28295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:08.064357 28295 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:08.064435 28295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:08.064525 28295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.71
I20240501 14:09:08.068111 28295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.23.191.71
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--never_fsync=true
--heap_profile_path=/tmp/kudu.28295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.71
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:08.069090 28295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:08.070250 28295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:08.078363 28303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:08.078658 28301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:08.079205 28300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:08.079732 28295 server_base.cc:1034] running on GCE node
I20240501 14:09:08.080106 28295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:08.080669 28295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:08.081833 28295 hybrid_clock.cc:648] HybridClock initialized: now 1714572548081804 us; error 42 us; skew 500 ppm
I20240501 14:09:08.083632 28295 webserver.cc:460] Webserver started at http://127.23.191.71:42275/ using document root <none> and password file <none>
I20240501 14:09:08.084174 28295 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:08.084230 28295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:08.084411 28295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:08.085680 27637 heartbeater.cc:503] Master 127.23.191.126:38485 was elected leader, sending a full tablet report...
I20240501 14:09:08.086190 28295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7"
format_stamp: "Formatted at 2024-05-01 14:09:08 on dist-test-slave-c4g3"
I20240501 14:09:08.086664 28295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7"
format_stamp: "Formatted at 2024-05-01 14:09:08 on dist-test-slave-c4g3"
I20240501 14:09:08.090175 28295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:08.092267 28309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:08.093289 28295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:08.093386 28295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7"
format_stamp: "Formatted at 2024-05-01 14:09:08 on dist-test-slave-c4g3"
I20240501 14:09:08.093472 28295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:08.106216 28295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:08.106889 28295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:08.107050 28295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:08.107578 28295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:08.108441 28295 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:08.108486 28295 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:08.108522 28295 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:08.108544 28295 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:08.116391 28295 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.71:33963
I20240501 14:09:08.116412 28422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.71:33963 every 8 connection(s)
I20240501 14:09:08.117317 28295 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:09:08.120064 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 28295
I20240501 14:09:08.120178 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:09:08.125830 28423 heartbeater.cc:347] Connected to a master server at 127.23.191.126:38485
I20240501 14:09:08.126024 28423 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:08.126559 28423 heartbeater.cc:511] Master 127.23.191.126:38485 requested a full tablet report, sending...
I20240501 14:09:08.127507 27449 ts_manager.cc:194] Registered new tserver with Master: ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71:33963)
I20240501 14:09:08.128046 27449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.71:58379
I20240501 14:09:08.134359 24317 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 14:09:08.142558 27449 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:09:08.159901 27964 tablet_service.cc:1467] Processing CreateTablet for tablet 78612cef5af34df3b33567744725af28 (DEFAULT_TABLE table=TestTable [id=b058b108878d4766b880a2ddcd9bfa90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:08.160408 28226 tablet_service.cc:1467] Processing CreateTablet for tablet 78612cef5af34df3b33567744725af28 (DEFAULT_TABLE table=TestTable [id=b058b108878d4766b880a2ddcd9bfa90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:08.160935 27964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78612cef5af34df3b33567744725af28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:08.161303 28226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78612cef5af34df3b33567744725af28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:08.166031 28440 tablet_bootstrap.cc:492] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17: Bootstrap starting.
I20240501 14:09:08.166522 28439 tablet_bootstrap.cc:492] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap starting.
I20240501 14:09:08.168037 28440 tablet_bootstrap.cc:654] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:08.168124 28439 tablet_bootstrap.cc:654] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:08.168706 28440 log.cc:824] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:08.168723 28439 log.cc:824] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:08.170542 28440 tablet_bootstrap.cc:492] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17: No bootstrap required, opened a new log
I20240501 14:09:08.170542 28439 tablet_bootstrap.cc:492] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5: No bootstrap required, opened a new log
I20240501 14:09:08.170728 28440 ts_tablet_manager.cc:1367] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:09:08.170728 28439 ts_tablet_manager.cc:1367] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:08.174110 28440 raft_consensus.cc:348] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } }
I20240501 14:09:08.174391 28440 raft_consensus.cc:374] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:08.174469 28440 raft_consensus.cc:729] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c37ce2bb5d844c09969e644d463cf17, State: Initialized, Role: FOLLOWER
I20240501 14:09:08.174393 28439 raft_consensus.cc:348] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } }
I20240501 14:09:08.174572 28439 raft_consensus.cc:374] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:08.174623 28439 raft_consensus.cc:729] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3625e18e2ea14d43aa3b86b014fd89f5, State: Initialized, Role: FOLLOWER
I20240501 14:09:08.174978 28440 consensus_queue.cc:260] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } }
I20240501 14:09:08.174978 28439 consensus_queue.cc:260] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } }
I20240501 14:09:08.175920 28440 ts_tablet_manager.cc:1398] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:09:08.175920 28439 ts_tablet_manager.cc:1398] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:09:08.176174 28292 heartbeater.cc:503] Master 127.23.191.126:38485 was elected leader, sending a full tablet report...
I20240501 14:09:08.176445 28030 heartbeater.cc:503] Master 127.23.191.126:38485 was elected leader, sending a full tablet report...
W20240501 14:09:08.199633 28293 tablet.cc:2369] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:08.243597 28444 raft_consensus.cc:482] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:08.243827 28444 raft_consensus.cc:504] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } }
I20240501 14:09:08.244872 28444 leader_election.cc:288] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68:43375)
I20240501 14:09:08.248224 27984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78612cef5af34df3b33567744725af28" candidate_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c37ce2bb5d844c09969e644d463cf17" is_pre_election: true
I20240501 14:09:08.248551 27984 raft_consensus.cc:2454] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3625e18e2ea14d43aa3b86b014fd89f5 in term 0.
I20240501 14:09:08.249068 28181 leader_election.cc:302] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3625e18e2ea14d43aa3b86b014fd89f5, 4c37ce2bb5d844c09969e644d463cf17; no voters:
I20240501 14:09:08.249326 28444 raft_consensus.cc:2781] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:08.249418 28444 raft_consensus.cc:482] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:08.249456 28444 raft_consensus.cc:3037] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:08.252458 28444 raft_consensus.cc:504] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } }
I20240501 14:09:08.252782 28444 leader_election.cc:288] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [CANDIDATE]: Term 1 election: Requested vote from peers 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68:43375)
I20240501 14:09:08.253610 27984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78612cef5af34df3b33567744725af28" candidate_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c37ce2bb5d844c09969e644d463cf17"
I20240501 14:09:08.253749 27984 raft_consensus.cc:3037] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:08.254622 27984 raft_consensus.cc:2454] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3625e18e2ea14d43aa3b86b014fd89f5 in term 1.
I20240501 14:09:08.254990 28181 leader_election.cc:302] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3625e18e2ea14d43aa3b86b014fd89f5, 4c37ce2bb5d844c09969e644d463cf17; no voters:
I20240501 14:09:08.255215 28444 raft_consensus.cc:2781] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:08.255452 28444 raft_consensus.cc:686] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEADER]: Becoming Leader. State: Replica: 3625e18e2ea14d43aa3b86b014fd89f5, State: Running, Role: LEADER
I20240501 14:09:08.255802 28444 consensus_queue.cc:237] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } }
I20240501 14:09:08.258946 27449 catalog_manager.cc:5484] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70). New cstate: current_term: 1 leader_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:08.261806 27768 heartbeater.cc:503] Master 127.23.191.126:38485 was elected leader, sending a full tablet report...
I20240501 14:09:08.278092 24317 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 14:09:08.279481 24317 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c9cc3f1efaf44a4895d3109beabe5312 to finish bootstrapping
I20240501 14:09:08.285214 24317 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fec89cd1e3ba491eab76811aae4af170 to finish bootstrapping
I20240501 14:09:08.290318 24317 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d1f8294a92e74484ab4a653a4ecf84e5 to finish bootstrapping
I20240501 14:09:08.295753 24317 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4c37ce2bb5d844c09969e644d463cf17 to finish bootstrapping
I20240501 14:09:08.300415 24317 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ae4fd219bcae42e3969310879588dc32 to finish bootstrapping
I20240501 14:09:08.305543 24317 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3625e18e2ea14d43aa3b86b014fd89f5 to finish bootstrapping
I20240501 14:09:08.310024 24317 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ff8b0a555ccd48bcb11dca38d3b2f2f7 to finish bootstrapping
I20240501 14:09:08.315187 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 27902
I20240501 14:09:08.320209 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 28033
I20240501 14:09:08.325121 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 28164
I20240501 14:09:08.330109 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 28295
I20240501 14:09:08.446508 27899 heartbeater.cc:503] Master 127.23.191.126:38485 was elected leader, sending a full tablet report...
I20240501 14:09:12.086058 27449 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:12.099413 27571 tablet_service.cc:1467] Processing CreateTablet for tablet e06d6b042b604fd19f19c44c3161beb8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ab50a0e1b4c24c32aba260261f572686]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:12.099413 27702 tablet_service.cc:1467] Processing CreateTablet for tablet e0d87c63b9f64212a3b451bb4140979e (DEFAULT_TABLE table=rebalance_test_table_0 [id=ab50a0e1b4c24c32aba260261f572686]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:12.099413 27570 tablet_service.cc:1467] Processing CreateTablet for tablet e0d87c63b9f64212a3b451bb4140979e (DEFAULT_TABLE table=rebalance_test_table_0 [id=ab50a0e1b4c24c32aba260261f572686]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:12.100458 27571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06d6b042b604fd19f19c44c3161beb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.100457 27702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d87c63b9f64212a3b451bb4140979e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.100363 27833 tablet_service.cc:1467] Processing CreateTablet for tablet e06d6b042b604fd19f19c44c3161beb8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ab50a0e1b4c24c32aba260261f572686]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:12.100652 27832 tablet_service.cc:1467] Processing CreateTablet for tablet c717dce4871e459390c508e275c52b31 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ab50a0e1b4c24c32aba260261f572686]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:12.101233 27832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c717dce4871e459390c508e275c52b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.101552 27833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06d6b042b604fd19f19c44c3161beb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.103394 27701 tablet_service.cc:1467] Processing CreateTablet for tablet c717dce4871e459390c508e275c52b31 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ab50a0e1b4c24c32aba260261f572686]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:12.103689 27701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c717dce4871e459390c508e275c52b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.105301 27570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d87c63b9f64212a3b451bb4140979e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.106894 28463 tablet_bootstrap.cc:492] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312: Bootstrap starting.
I20240501 14:09:12.107431 28464 tablet_bootstrap.cc:492] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170: Bootstrap starting.
I20240501 14:09:12.109331 28463 tablet_bootstrap.cc:654] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:12.109424 28465 tablet_bootstrap.cc:492] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5: Bootstrap starting.
I20240501 14:09:12.109458 28464 tablet_bootstrap.cc:654] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:12.110076 28464 log.cc:824] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:12.110134 28463 log.cc:824] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:12.111239 28465 tablet_bootstrap.cc:654] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:12.111631 28464 tablet_bootstrap.cc:492] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170: No bootstrap required, opened a new log
I20240501 14:09:12.111788 28464 ts_tablet_manager.cc:1367] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:12.111837 28465 log.cc:824] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:12.113293 28465 tablet_bootstrap.cc:492] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5: No bootstrap required, opened a new log
I20240501 14:09:12.113456 28465 ts_tablet_manager.cc:1367] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:12.113981 28463 tablet_bootstrap.cc:492] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312: No bootstrap required, opened a new log
I20240501 14:09:12.114141 28463 ts_tablet_manager.cc:1367] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:09:12.115371 28464 raft_consensus.cc:348] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } }
I20240501 14:09:12.115597 28464 raft_consensus.cc:374] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:12.115653 28464 raft_consensus.cc:729] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Initialized, Role: FOLLOWER
I20240501 14:09:12.116122 28464 consensus_queue.cc:260] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } }
I20240501 14:09:12.116492 28465 raft_consensus.cc:348] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.116683 28465 raft_consensus.cc:374] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:12.116726 28465 raft_consensus.cc:729] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Initialized, Role: FOLLOWER
I20240501 14:09:12.117065 28464 ts_tablet_manager.cc:1398] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:09:12.117177 28465 consensus_queue.cc:260] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.117308 28464 tablet_bootstrap.cc:492] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170: Bootstrap starting.
I20240501 14:09:12.117678 28463 raft_consensus.cc:348] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.117871 28463 raft_consensus.cc:374] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:12.117920 28463 raft_consensus.cc:729] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Initialized, Role: FOLLOWER
I20240501 14:09:12.117970 28465 ts_tablet_manager.cc:1398] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:09:12.118292 28465 tablet_bootstrap.cc:492] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5: Bootstrap starting.
I20240501 14:09:12.118317 28464 tablet_bootstrap.cc:654] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:12.118463 28463 consensus_queue.cc:260] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.119194 28465 tablet_bootstrap.cc:654] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:12.119314 28463 ts_tablet_manager.cc:1398] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:09:12.119665 28463 tablet_bootstrap.cc:492] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312: Bootstrap starting.
I20240501 14:09:12.120265 28465 tablet_bootstrap.cc:492] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5: No bootstrap required, opened a new log
I20240501 14:09:12.120355 28465 ts_tablet_manager.cc:1367] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:12.120640 28463 tablet_bootstrap.cc:654] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:12.120918 28465 raft_consensus.cc:348] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.121026 28465 raft_consensus.cc:374] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:12.121064 28465 raft_consensus.cc:729] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Initialized, Role: FOLLOWER
I20240501 14:09:12.121248 28465 consensus_queue.cc:260] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.121616 28465 ts_tablet_manager.cc:1398] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:12.122051 28463 tablet_bootstrap.cc:492] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312: No bootstrap required, opened a new log
I20240501 14:09:12.122134 28463 ts_tablet_manager.cc:1367] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:12.122678 28463 raft_consensus.cc:348] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } }
I20240501 14:09:12.122785 28463 raft_consensus.cc:374] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:12.122813 28463 raft_consensus.cc:729] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Initialized, Role: FOLLOWER
I20240501 14:09:12.122959 28463 consensus_queue.cc:260] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } }
I20240501 14:09:12.123242 28463 ts_tablet_manager.cc:1398] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:12.123369 28464 tablet_bootstrap.cc:492] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170: No bootstrap required, opened a new log
I20240501 14:09:12.123442 28464 ts_tablet_manager.cc:1367] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:12.123950 28464 raft_consensus.cc:348] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.124068 28464 raft_consensus.cc:374] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:12.124100 28464 raft_consensus.cc:729] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Initialized, Role: FOLLOWER
I20240501 14:09:12.124225 28464 consensus_queue.cc:260] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.124472 28464 ts_tablet_manager.cc:1398] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 14:09:12.186532 27900 tablet.cc:2369] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:12.230747 28475 raft_consensus.cc:482] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:12.230978 28475 raft_consensus.cc:504] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.232285 28475 leader_election.cc:288] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:12.235881 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06d6b042b604fd19f19c44c3161beb8" candidate_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312" is_pre_election: true
I20240501 14:09:12.236200 27591 raft_consensus.cc:2454] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f8294a92e74484ab4a653a4ecf84e5 in term 0.
I20240501 14:09:12.236649 27786 leader_election.cc:302] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, d1f8294a92e74484ab4a653a4ecf84e5; no voters:
I20240501 14:09:12.236963 28475 raft_consensus.cc:2781] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:12.237078 28475 raft_consensus.cc:482] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:12.237112 28475 raft_consensus.cc:3037] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:12.238016 28475 raft_consensus.cc:504] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.238324 28475 leader_election.cc:288] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 election: Requested vote from peers c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:12.238691 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06d6b042b604fd19f19c44c3161beb8" candidate_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
I20240501 14:09:12.238818 27591 raft_consensus.cc:3037] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:12.239706 27591 raft_consensus.cc:2454] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f8294a92e74484ab4a653a4ecf84e5 in term 1.
I20240501 14:09:12.240015 27786 leader_election.cc:302] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, d1f8294a92e74484ab4a653a4ecf84e5; no voters:
I20240501 14:09:12.240211 28475 raft_consensus.cc:2781] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:12.240306 28475 raft_consensus.cc:686] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Becoming Leader. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Running, Role: LEADER
I20240501 14:09:12.240644 28475 consensus_queue.cc:237] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.243844 27448 catalog_manager.cc:5484] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67). New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:12.254163 27769 tablet.cc:2369] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:12.318293 28469 raft_consensus.cc:482] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:12.318490 28469 raft_consensus.cc:504] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.319470 28469 leader_election.cc:288] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:12.323125 27853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c717dce4871e459390c508e275c52b31" candidate_uuid: "fec89cd1e3ba491eab76811aae4af170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" is_pre_election: true
I20240501 14:09:12.323351 27853 raft_consensus.cc:2454] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec89cd1e3ba491eab76811aae4af170 in term 0.
I20240501 14:09:12.323729 27658 leader_election.cc:302] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d1f8294a92e74484ab4a653a4ecf84e5, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:12.323972 28469 raft_consensus.cc:2781] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:12.324054 28469 raft_consensus.cc:482] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:12.324100 28469 raft_consensus.cc:3037] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:12.325301 28469 raft_consensus.cc:504] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.325590 28469 leader_election.cc:288] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 1 election: Requested vote from peers d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:12.325978 27853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c717dce4871e459390c508e275c52b31" candidate_uuid: "fec89cd1e3ba491eab76811aae4af170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f8294a92e74484ab4a653a4ecf84e5"
I20240501 14:09:12.326093 27853 raft_consensus.cc:3037] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:09:12.326444 27638 tablet.cc:2369] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:12.326983 27853 raft_consensus.cc:2454] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec89cd1e3ba491eab76811aae4af170 in term 1.
I20240501 14:09:12.327265 27658 leader_election.cc:302] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d1f8294a92e74484ab4a653a4ecf84e5, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:12.327422 28469 raft_consensus.cc:2781] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:12.327633 28469 raft_consensus.cc:686] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Becoming Leader. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Running, Role: LEADER
I20240501 14:09:12.327929 28469 consensus_queue.cc:237] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.331022 27448 catalog_manager.cc:5484] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec89cd1e3ba491eab76811aae4af170 (127.23.191.66). New cstate: current_term: 1 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:12.383134 28471 raft_consensus.cc:482] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:12.383287 28471 raft_consensus.cc:504] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } }
I20240501 14:09:12.384265 28471 leader_election.cc:288] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:12.387759 27722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d87c63b9f64212a3b451bb4140979e" candidate_uuid: "c9cc3f1efaf44a4895d3109beabe5312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec89cd1e3ba491eab76811aae4af170" is_pre_election: true
I20240501 14:09:12.387964 27722 raft_consensus.cc:2454] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9cc3f1efaf44a4895d3109beabe5312 in term 0.
I20240501 14:09:12.388367 27525 leader_election.cc:302] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:12.388607 28471 raft_consensus.cc:2781] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:12.388676 28471 raft_consensus.cc:482] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:12.388702 28471 raft_consensus.cc:3037] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:12.389565 28471 raft_consensus.cc:504] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } }
I20240501 14:09:12.389851 28471 leader_election.cc:288] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 election: Requested vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:12.390195 27722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d87c63b9f64212a3b451bb4140979e" candidate_uuid: "c9cc3f1efaf44a4895d3109beabe5312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
I20240501 14:09:12.390311 27722 raft_consensus.cc:3037] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:12.391072 27722 raft_consensus.cc:2454] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9cc3f1efaf44a4895d3109beabe5312 in term 1.
I20240501 14:09:12.391355 27525 leader_election.cc:302] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:12.391513 28471 raft_consensus.cc:2781] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:12.391726 28471 raft_consensus.cc:686] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Becoming Leader. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Running, Role: LEADER
I20240501 14:09:12.392011 28471 consensus_queue.cc:237] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } }
I20240501 14:09:12.394878 27448 catalog_manager.cc:5484] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:12.562610 28484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:12.562852 28484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:12.565845 28484 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 14:09:12.648516 28475 consensus_queue.cc:1035] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:12.667171 27853 raft_consensus.cc:1261] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:12.667802 28469 consensus_queue.cc:1035] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:12.667969 27722 raft_consensus.cc:1261] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:12.668535 28471 consensus_queue.cc:1035] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:12.675206 28525 mvcc.cc:205] Tried to move back new op lower bound from 7022889175718522880 to 7022889174337085440. Current Snapshot: MvccSnapshot[applied={T|T < 7022889175718522880}]
I20240501 14:09:12.679731 28521 mvcc.cc:205] Tried to move back new op lower bound from 7022889175718522880 to 7022889174337085440. Current Snapshot: MvccSnapshot[applied={T|T < 7022889175718522880}]
I20240501 14:09:12.685526 28519 mvcc.cc:205] Tried to move back new op lower bound from 7022889175718666240 to 7022889174599630848. Current Snapshot: MvccSnapshot[applied={T|T < 7022889175718666240}]
INSERT report
rows total: 2000
time total: 105.406 ms
time per row: 0.052703 ms
I20240501 14:09:12.714020 27447 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:12.719353 27570 tablet_service.cc:1467] Processing CreateTablet for tablet b54ea6caa4f6448790b5456a2e5676c2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4f58d714ee014c97a0e588bae9a3e9c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:12.719394 27571 tablet_service.cc:1467] Processing CreateTablet for tablet ad72089b2cf54cd0b61ee8bb2c95975f (DEFAULT_TABLE table=rebalance_test_table_1 [id=4f58d714ee014c97a0e588bae9a3e9c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:12.719604 27701 tablet_service.cc:1467] Processing CreateTablet for tablet b54ea6caa4f6448790b5456a2e5676c2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4f58d714ee014c97a0e588bae9a3e9c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:12.719697 27570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54ea6caa4f6448790b5456a2e5676c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.719879 27701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54ea6caa4f6448790b5456a2e5676c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.719923 27571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad72089b2cf54cd0b61ee8bb2c95975f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.720026 27702 tablet_service.cc:1467] Processing CreateTablet for tablet fe7fe0171f2d477c91661a766239b9fa (DEFAULT_TABLE table=rebalance_test_table_1 [id=4f58d714ee014c97a0e588bae9a3e9c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:12.720188 27833 tablet_service.cc:1467] Processing CreateTablet for tablet ad72089b2cf54cd0b61ee8bb2c95975f (DEFAULT_TABLE table=rebalance_test_table_1 [id=4f58d714ee014c97a0e588bae9a3e9c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:12.720259 27702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7fe0171f2d477c91661a766239b9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.720427 27833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad72089b2cf54cd0b61ee8bb2c95975f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.722380 28537 tablet_bootstrap.cc:492] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312: Bootstrap starting.
I20240501 14:09:12.722606 27832 tablet_service.cc:1467] Processing CreateTablet for tablet fe7fe0171f2d477c91661a766239b9fa (DEFAULT_TABLE table=rebalance_test_table_1 [id=4f58d714ee014c97a0e588bae9a3e9c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:12.722821 27832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7fe0171f2d477c91661a766239b9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.723177 28537 tablet_bootstrap.cc:654] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:12.723737 28536 tablet_bootstrap.cc:492] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170: Bootstrap starting.
I20240501 14:09:12.724148 28537 tablet_bootstrap.cc:492] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312: No bootstrap required, opened a new log
I20240501 14:09:12.724247 28537 ts_tablet_manager.cc:1367] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:12.724412 28536 tablet_bootstrap.cc:654] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:12.724808 28537 raft_consensus.cc:348] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.724985 28537 raft_consensus.cc:374] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:12.725019 28537 raft_consensus.cc:729] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Initialized, Role: FOLLOWER
I20240501 14:09:12.725060 28538 tablet_bootstrap.cc:492] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5: Bootstrap starting.
I20240501 14:09:12.725167 28537 consensus_queue.cc:260] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.725518 28537 ts_tablet_manager.cc:1398] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:12.726097 28537 tablet_bootstrap.cc:492] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312: Bootstrap starting.
I20240501 14:09:12.726243 28538 tablet_bootstrap.cc:654] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:12.727110 28537 tablet_bootstrap.cc:654] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:12.727277 28538 tablet_bootstrap.cc:492] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5: No bootstrap required, opened a new log
I20240501 14:09:12.727310 28536 tablet_bootstrap.cc:492] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170: No bootstrap required, opened a new log
I20240501 14:09:12.727348 28538 ts_tablet_manager.cc:1367] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:12.727386 28536 ts_tablet_manager.cc:1367] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:09:12.727768 28538 raft_consensus.cc:348] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.727855 28538 raft_consensus.cc:374] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:12.727885 28538 raft_consensus.cc:729] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Initialized, Role: FOLLOWER
I20240501 14:09:12.728000 28538 consensus_queue.cc:260] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.728070 28537 tablet_bootstrap.cc:492] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312: No bootstrap required, opened a new log
I20240501 14:09:12.728056 28536 raft_consensus.cc:348] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:12.728135 28537 ts_tablet_manager.cc:1367] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:12.728165 28536 raft_consensus.cc:374] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:12.728200 28536 raft_consensus.cc:729] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Initialized, Role: FOLLOWER
I20240501 14:09:12.728260 28538 ts_tablet_manager.cc:1398] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:12.728307 28536 consensus_queue.cc:260] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:12.728415 28538 tablet_bootstrap.cc:492] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5: Bootstrap starting.
I20240501 14:09:12.728545 28536 ts_tablet_manager.cc:1398] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:12.728518 28537 raft_consensus.cc:348] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:12.728596 28537 raft_consensus.cc:374] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:12.728624 28537 raft_consensus.cc:729] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Initialized, Role: FOLLOWER
I20240501 14:09:12.728700 28536 tablet_bootstrap.cc:492] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170: Bootstrap starting.
I20240501 14:09:12.728740 28537 consensus_queue.cc:260] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:12.728983 28537 ts_tablet_manager.cc:1398] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:12.729408 28538 tablet_bootstrap.cc:654] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:12.729540 28536 tablet_bootstrap.cc:654] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:12.730629 28538 tablet_bootstrap.cc:492] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5: No bootstrap required, opened a new log
I20240501 14:09:12.730648 28536 tablet_bootstrap.cc:492] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170: No bootstrap required, opened a new log
I20240501 14:09:12.730708 28538 ts_tablet_manager.cc:1367] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:12.730717 28536 ts_tablet_manager.cc:1367] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:12.731593 28538 raft_consensus.cc:348] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.731693 28538 raft_consensus.cc:374] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:12.731735 28538 raft_consensus.cc:729] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Initialized, Role: FOLLOWER
I20240501 14:09:12.731740 28536 raft_consensus.cc:348] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.731829 28536 raft_consensus.cc:374] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:12.731868 28536 raft_consensus.cc:729] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Initialized, Role: FOLLOWER
I20240501 14:09:12.731854 28538 consensus_queue.cc:260] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.731977 28536 consensus_queue.cc:260] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.732213 28538 ts_tablet_manager.cc:1398] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:12.732223 28536 ts_tablet_manager.cc:1398] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:12.760604 28471 raft_consensus.cc:482] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:12.760744 28471 raft_consensus.cc:504] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.761150 28471 leader_election.cc:288] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:12.764366 27853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad72089b2cf54cd0b61ee8bb2c95975f" candidate_uuid: "c9cc3f1efaf44a4895d3109beabe5312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" is_pre_election: true
I20240501 14:09:12.764501 27853 raft_consensus.cc:2454] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9cc3f1efaf44a4895d3109beabe5312 in term 0.
I20240501 14:09:12.764798 27527 leader_election.cc:302] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, d1f8294a92e74484ab4a653a4ecf84e5; no voters:
I20240501 14:09:12.764987 28471 raft_consensus.cc:2781] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:12.765044 28471 raft_consensus.cc:482] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:12.765072 28471 raft_consensus.cc:3037] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:12.765812 28471 raft_consensus.cc:504] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.766070 28471 leader_election.cc:288] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 election: Requested vote from peers d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:12.766398 27853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad72089b2cf54cd0b61ee8bb2c95975f" candidate_uuid: "c9cc3f1efaf44a4895d3109beabe5312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f8294a92e74484ab4a653a4ecf84e5"
I20240501 14:09:12.766523 27853 raft_consensus.cc:3037] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:12.767218 27853 raft_consensus.cc:2454] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9cc3f1efaf44a4895d3109beabe5312 in term 1.
I20240501 14:09:12.767467 27527 leader_election.cc:302] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, d1f8294a92e74484ab4a653a4ecf84e5; no voters:
I20240501 14:09:12.767614 28471 raft_consensus.cc:2781] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:12.767688 28471 raft_consensus.cc:686] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Becoming Leader. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Running, Role: LEADER
I20240501 14:09:12.767808 28471 consensus_queue.cc:237] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.769356 27446 catalog_manager.cc:5484] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:12.772162 28471 raft_consensus.cc:482] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:12.772238 28471 raft_consensus.cc:504] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:12.772490 28471 leader_election.cc:288] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:12.772904 27722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54ea6caa4f6448790b5456a2e5676c2" candidate_uuid: "c9cc3f1efaf44a4895d3109beabe5312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec89cd1e3ba491eab76811aae4af170" is_pre_election: true
I20240501 14:09:12.773028 27722 raft_consensus.cc:2454] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9cc3f1efaf44a4895d3109beabe5312 in term 0.
I20240501 14:09:12.773275 27525 leader_election.cc:302] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:12.773414 28471 raft_consensus.cc:2781] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:12.773469 28471 raft_consensus.cc:482] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:12.773505 28471 raft_consensus.cc:3037] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:12.774267 28471 raft_consensus.cc:504] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:12.774518 28471 leader_election.cc:288] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 election: Requested vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:12.774852 27722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54ea6caa4f6448790b5456a2e5676c2" candidate_uuid: "c9cc3f1efaf44a4895d3109beabe5312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
I20240501 14:09:12.774955 27722 raft_consensus.cc:3037] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:12.775815 27722 raft_consensus.cc:2454] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9cc3f1efaf44a4895d3109beabe5312 in term 1.
I20240501 14:09:12.776103 27525 leader_election.cc:302] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:12.776269 28471 raft_consensus.cc:2781] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:12.776368 28471 raft_consensus.cc:686] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Becoming Leader. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Running, Role: LEADER
I20240501 14:09:12.776477 28471 consensus_queue.cc:237] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:12.778007 27446 catalog_manager.cc:5484] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:12.842383 28470 raft_consensus.cc:482] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:12.842537 28470 raft_consensus.cc:504] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.843004 28470 leader_election.cc:288] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:12.846395 27722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7fe0171f2d477c91661a766239b9fa" candidate_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec89cd1e3ba491eab76811aae4af170" is_pre_election: true
I20240501 14:09:12.846554 27722 raft_consensus.cc:2454] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f8294a92e74484ab4a653a4ecf84e5 in term 0.
I20240501 14:09:12.846825 27787 leader_election.cc:302] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d1f8294a92e74484ab4a653a4ecf84e5, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:12.846997 28470 raft_consensus.cc:2781] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:12.847056 28470 raft_consensus.cc:482] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:12.847084 28470 raft_consensus.cc:3037] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:12.847894 28470 raft_consensus.cc:504] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.848161 28470 leader_election.cc:288] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 election: Requested vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:12.848510 27722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7fe0171f2d477c91661a766239b9fa" candidate_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
I20240501 14:09:12.848619 27722 raft_consensus.cc:3037] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:12.849334 27722 raft_consensus.cc:2454] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f8294a92e74484ab4a653a4ecf84e5 in term 1.
I20240501 14:09:12.849579 27787 leader_election.cc:302] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d1f8294a92e74484ab4a653a4ecf84e5, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:12.849728 28470 raft_consensus.cc:2781] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:12.849803 28470 raft_consensus.cc:686] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Becoming Leader. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Running, Role: LEADER
I20240501 14:09:12.849933 28470 consensus_queue.cc:237] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:12.851435 27446 catalog_manager.cc:5484] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67). New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:12.981149 28549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:12.981391 28549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:12.984475 28549 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:09:13.057395 27722 raft_consensus.cc:1261] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:13.057920 28470 consensus_queue.cc:1035] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:13.058662 27721 raft_consensus.cc:1261] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:13.061713 27853 raft_consensus.cc:1261] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:13.062073 28471 consensus_queue.cc:1035] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:13.062762 28471 consensus_queue.cc:1035] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 84.3858 ms
time per row: 0.0421929 ms
I20240501 14:09:13.107592 27449 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:13.113137 27570 tablet_service.cc:1467] Processing CreateTablet for tablet ea2e1d9ab9a54d52b89c72c4ef442ce5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8615b34ff7ef42beb6766dea8a59f98a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:13.113456 27570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2e1d9ab9a54d52b89c72c4ef442ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:13.113432 27702 tablet_service.cc:1467] Processing CreateTablet for tablet ea2e1d9ab9a54d52b89c72c4ef442ce5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8615b34ff7ef42beb6766dea8a59f98a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:13.113675 27702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2e1d9ab9a54d52b89c72c4ef442ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:13.113958 27701 tablet_service.cc:1467] Processing CreateTablet for tablet 8a5ec8adfb714bf98dcf3279b9df2148 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8615b34ff7ef42beb6766dea8a59f98a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:13.114181 27701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5ec8adfb714bf98dcf3279b9df2148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:13.115540 28537 tablet_bootstrap.cc:492] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312: Bootstrap starting.
I20240501 14:09:13.115779 27832 tablet_service.cc:1467] Processing CreateTablet for tablet ca6f19b8716843ccaabf666651584928 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8615b34ff7ef42beb6766dea8a59f98a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:13.116051 27832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6f19b8716843ccaabf666651584928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:13.116382 28537 tablet_bootstrap.cc:654] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:13.116364 27570 tablet_service.cc:1467] Processing CreateTablet for tablet ca6f19b8716843ccaabf666651584928 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8615b34ff7ef42beb6766dea8a59f98a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:13.116565 27570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6f19b8716843ccaabf666651584928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:13.115779 27833 tablet_service.cc:1467] Processing CreateTablet for tablet 8a5ec8adfb714bf98dcf3279b9df2148 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8615b34ff7ef42beb6766dea8a59f98a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:13.116858 27833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5ec8adfb714bf98dcf3279b9df2148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:13.116942 28536 tablet_bootstrap.cc:492] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170: Bootstrap starting.
I20240501 14:09:13.117738 28536 tablet_bootstrap.cc:654] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:13.118067 28538 tablet_bootstrap.cc:492] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5: Bootstrap starting.
I20240501 14:09:13.118893 28538 tablet_bootstrap.cc:654] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:13.119192 28536 tablet_bootstrap.cc:492] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170: No bootstrap required, opened a new log
I20240501 14:09:13.119278 28536 ts_tablet_manager.cc:1367] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:13.119434 28537 tablet_bootstrap.cc:492] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312: No bootstrap required, opened a new log
I20240501 14:09:13.119524 28537 ts_tablet_manager.cc:1367] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:09:13.119710 28536 raft_consensus.cc:348] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.119835 28536 raft_consensus.cc:374] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:13.119869 28536 raft_consensus.cc:729] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Initialized, Role: FOLLOWER
I20240501 14:09:13.119864 28537 raft_consensus.cc:348] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.119946 28538 tablet_bootstrap.cc:492] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5: No bootstrap required, opened a new log
I20240501 14:09:13.119968 28537 raft_consensus.cc:374] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:13.120000 28537 raft_consensus.cc:729] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Initialized, Role: FOLLOWER
I20240501 14:09:13.120008 28538 ts_tablet_manager.cc:1367] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:13.120003 28536 consensus_queue.cc:260] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.120113 28537 consensus_queue.cc:260] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.120283 28536 ts_tablet_manager.cc:1398] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:13.120349 28537 ts_tablet_manager.cc:1398] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:13.120393 28538 raft_consensus.cc:348] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:13.120461 28536 tablet_bootstrap.cc:492] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170: Bootstrap starting.
I20240501 14:09:13.120481 28538 raft_consensus.cc:374] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:13.120510 28538 raft_consensus.cc:729] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Initialized, Role: FOLLOWER
I20240501 14:09:13.120515 28537 tablet_bootstrap.cc:492] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312: Bootstrap starting.
I20240501 14:09:13.120615 28538 consensus_queue.cc:260] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:13.120898 28538 ts_tablet_manager.cc:1398] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:13.121055 28538 tablet_bootstrap.cc:492] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5: Bootstrap starting.
I20240501 14:09:13.121275 28537 tablet_bootstrap.cc:654] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:13.121352 28536 tablet_bootstrap.cc:654] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:13.121871 28538 tablet_bootstrap.cc:654] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:13.122381 28537 tablet_bootstrap.cc:492] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312: No bootstrap required, opened a new log
I20240501 14:09:13.122642 28537 ts_tablet_manager.cc:1367] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:13.122838 28538 tablet_bootstrap.cc:492] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5: No bootstrap required, opened a new log
I20240501 14:09:13.122921 28538 ts_tablet_manager.cc:1367] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:13.123239 28537 raft_consensus.cc:348] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:13.123469 28538 raft_consensus.cc:348] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:13.123560 28538 raft_consensus.cc:374] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:13.123591 28538 raft_consensus.cc:729] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Initialized, Role: FOLLOWER
I20240501 14:09:13.123703 28538 consensus_queue.cc:260] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:13.123940 28538 ts_tablet_manager.cc:1398] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:13.124214 28536 tablet_bootstrap.cc:492] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170: No bootstrap required, opened a new log
I20240501 14:09:13.124270 28536 ts_tablet_manager.cc:1367] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:09:13.124753 28536 raft_consensus.cc:348] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:13.124864 28536 raft_consensus.cc:374] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:13.124902 28536 raft_consensus.cc:729] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Initialized, Role: FOLLOWER
I20240501 14:09:13.125022 28536 consensus_queue.cc:260] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:13.125272 28536 ts_tablet_manager.cc:1398] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:13.125464 28537 raft_consensus.cc:374] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:13.125521 28537 raft_consensus.cc:729] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Initialized, Role: FOLLOWER
I20240501 14:09:13.125630 28537 consensus_queue.cc:260] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:13.125888 28537 ts_tablet_manager.cc:1398] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240501 14:09:13.201527 28470 raft_consensus.cc:482] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:13.201661 28470 raft_consensus.cc:504] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:13.201962 28470 leader_election.cc:288] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:13.202404 27722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5ec8adfb714bf98dcf3279b9df2148" candidate_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec89cd1e3ba491eab76811aae4af170" is_pre_election: true
I20240501 14:09:13.202539 27722 raft_consensus.cc:2454] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f8294a92e74484ab4a653a4ecf84e5 in term 0.
I20240501 14:09:13.202788 27787 leader_election.cc:302] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d1f8294a92e74484ab4a653a4ecf84e5, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:13.202991 28470 raft_consensus.cc:2781] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:13.203056 28470 raft_consensus.cc:482] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:13.203085 28470 raft_consensus.cc:3037] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:13.203853 28470 raft_consensus.cc:504] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:13.204135 28470 leader_election.cc:288] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 election: Requested vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:13.204495 27722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5ec8adfb714bf98dcf3279b9df2148" candidate_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
I20240501 14:09:13.204630 27722 raft_consensus.cc:3037] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:13.205389 27722 raft_consensus.cc:2454] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f8294a92e74484ab4a653a4ecf84e5 in term 1.
I20240501 14:09:13.205677 27787 leader_election.cc:302] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d1f8294a92e74484ab4a653a4ecf84e5, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:13.205932 28470 raft_consensus.cc:2781] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:13.206023 28470 raft_consensus.cc:686] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Becoming Leader. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Running, Role: LEADER
I20240501 14:09:13.206153 28470 consensus_queue.cc:237] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:13.207656 27443 catalog_manager.cc:5484] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67). New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:13.311312 28482 raft_consensus.cc:482] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:13.311451 28482 raft_consensus.cc:504] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:13.311826 28482 leader_election.cc:288] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:13.312304 27853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6f19b8716843ccaabf666651584928" candidate_uuid: "c9cc3f1efaf44a4895d3109beabe5312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" is_pre_election: true
I20240501 14:09:13.312438 27853 raft_consensus.cc:2454] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9cc3f1efaf44a4895d3109beabe5312 in term 0.
I20240501 14:09:13.312722 27527 leader_election.cc:302] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, d1f8294a92e74484ab4a653a4ecf84e5; no voters:
I20240501 14:09:13.312963 28482 raft_consensus.cc:2781] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:13.313032 28482 raft_consensus.cc:482] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:13.313113 28482 raft_consensus.cc:3037] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:13.314314 28482 raft_consensus.cc:504] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:13.314587 28482 leader_election.cc:288] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 election: Requested vote from peers d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:13.314919 27853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6f19b8716843ccaabf666651584928" candidate_uuid: "c9cc3f1efaf44a4895d3109beabe5312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f8294a92e74484ab4a653a4ecf84e5"
I20240501 14:09:13.315029 27853 raft_consensus.cc:3037] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:13.315752 27853 raft_consensus.cc:2454] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9cc3f1efaf44a4895d3109beabe5312 in term 1.
I20240501 14:09:13.316035 27527 leader_election.cc:302] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, d1f8294a92e74484ab4a653a4ecf84e5; no voters:
I20240501 14:09:13.316213 28482 raft_consensus.cc:2781] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:13.316306 28482 raft_consensus.cc:686] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Becoming Leader. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Running, Role: LEADER
I20240501 14:09:13.316442 28482 consensus_queue.cc:237] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:13.318110 27443 catalog_manager.cc:5484] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:13.482067 28482 raft_consensus.cc:482] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:13.482211 28482 raft_consensus.cc:504] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.482548 28482 leader_election.cc:288] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:13.482970 27722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2e1d9ab9a54d52b89c72c4ef442ce5" candidate_uuid: "c9cc3f1efaf44a4895d3109beabe5312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec89cd1e3ba491eab76811aae4af170" is_pre_election: true
I20240501 14:09:13.483119 27722 raft_consensus.cc:2454] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9cc3f1efaf44a4895d3109beabe5312 in term 0.
I20240501 14:09:13.483431 27525 leader_election.cc:302] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:13.483593 28482 raft_consensus.cc:2781] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:13.483652 28482 raft_consensus.cc:482] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:13.483676 28482 raft_consensus.cc:3037] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:13.484488 28482 raft_consensus.cc:504] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.484750 28482 leader_election.cc:288] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 election: Requested vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:13.485098 27722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2e1d9ab9a54d52b89c72c4ef442ce5" candidate_uuid: "c9cc3f1efaf44a4895d3109beabe5312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
I20240501 14:09:13.485222 27722 raft_consensus.cc:3037] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:13.485929 27722 raft_consensus.cc:2454] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9cc3f1efaf44a4895d3109beabe5312 in term 1.
I20240501 14:09:13.486215 27525 leader_election.cc:302] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:13.486366 28482 raft_consensus.cc:2781] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:13.486459 28482 raft_consensus.cc:686] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Becoming Leader. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Running, Role: LEADER
I20240501 14:09:13.486589 28482 consensus_queue.cc:237] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.488096 27443 catalog_manager.cc:5484] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:13.675457 28586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:13.675714 28586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:13.678804 28586 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:09:13.783214 27853 raft_consensus.cc:1261] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:13.783933 28471 consensus_queue.cc:1035] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:13.784863 27722 raft_consensus.cc:1261] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:13.785876 28471 consensus_queue.cc:1035] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:13.789119 27722 raft_consensus.cc:1261] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:13.789542 28470 consensus_queue.cc:1035] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.26 ms
time per row: 0.0531302 ms
I20240501 14:09:13.826290 27445 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:13.831336 27570 tablet_service.cc:1467] Processing CreateTablet for tablet 8cf4c3bcfc524e708092cd92d0622684 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78f3b25b930c48d7bc71a12bc14a9aaa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:13.831388 27701 tablet_service.cc:1467] Processing CreateTablet for tablet 8cf4c3bcfc524e708092cd92d0622684 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78f3b25b930c48d7bc71a12bc14a9aaa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:13.831679 27570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cf4c3bcfc524e708092cd92d0622684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:13.831678 27701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cf4c3bcfc524e708092cd92d0622684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:13.833452 28640 tablet_bootstrap.cc:492] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170: Bootstrap starting.
I20240501 14:09:13.833467 28641 tablet_bootstrap.cc:492] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312: Bootstrap starting.
I20240501 14:09:13.833916 27833 tablet_service.cc:1467] Processing CreateTablet for tablet 41d8c8dbc45b449583f0a1808c634855 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78f3b25b930c48d7bc71a12bc14a9aaa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:13.834188 27833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d8c8dbc45b449583f0a1808c634855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:13.834304 28641 tablet_bootstrap.cc:654] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:13.834563 27570 tablet_service.cc:1467] Processing CreateTablet for tablet 41d8c8dbc45b449583f0a1808c634855 (DEFAULT_TABLE table=rebalance_test_table_3 [id=78f3b25b930c48d7bc71a12bc14a9aaa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:13.834795 27570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d8c8dbc45b449583f0a1808c634855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:13.834810 28640 tablet_bootstrap.cc:654] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:13.835932 28640 tablet_bootstrap.cc:492] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170: No bootstrap required, opened a new log
I20240501 14:09:13.836021 28640 ts_tablet_manager.cc:1367] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:13.836380 28644 tablet_bootstrap.cc:492] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5: Bootstrap starting.
I20240501 14:09:13.836486 27832 tablet_service.cc:1467] Processing CreateTablet for tablet dcbe4acb729e48bb89d5a26cb211b2eb (DEFAULT_TABLE table=rebalance_test_table_3 [id=78f3b25b930c48d7bc71a12bc14a9aaa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:13.836632 28640 raft_consensus.cc:348] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.836728 27832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbe4acb729e48bb89d5a26cb211b2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:13.836751 28640 raft_consensus.cc:374] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:13.836787 28640 raft_consensus.cc:729] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Initialized, Role: FOLLOWER
I20240501 14:09:13.836959 28640 consensus_queue.cc:260] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.837497 28644 tablet_bootstrap.cc:654] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:13.837687 28640 ts_tablet_manager.cc:1398] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:13.838456 28644 tablet_bootstrap.cc:492] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5: No bootstrap required, opened a new log
I20240501 14:09:13.838575 28644 ts_tablet_manager.cc:1367] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:13.838994 28644 raft_consensus.cc:348] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } }
I20240501 14:09:13.839084 28644 raft_consensus.cc:374] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:13.839124 28644 raft_consensus.cc:729] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Initialized, Role: FOLLOWER
I20240501 14:09:13.839098 27701 tablet_service.cc:1467] Processing CreateTablet for tablet dcbe4acb729e48bb89d5a26cb211b2eb (DEFAULT_TABLE table=rebalance_test_table_3 [id=78f3b25b930c48d7bc71a12bc14a9aaa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:13.839241 28644 consensus_queue.cc:260] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } }
I20240501 14:09:13.839324 27701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbe4acb729e48bb89d5a26cb211b2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:13.839582 28644 ts_tablet_manager.cc:1398] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:13.839730 28644 tablet_bootstrap.cc:492] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5: Bootstrap starting.
I20240501 14:09:13.840526 28644 tablet_bootstrap.cc:654] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:13.841164 28640 tablet_bootstrap.cc:492] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170: Bootstrap starting.
I20240501 14:09:13.841822 28641 tablet_bootstrap.cc:492] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312: No bootstrap required, opened a new log
I20240501 14:09:13.841923 28640 tablet_bootstrap.cc:654] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:13.841934 28641 ts_tablet_manager.cc:1367] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:13.842152 28644 tablet_bootstrap.cc:492] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5: No bootstrap required, opened a new log
I20240501 14:09:13.842233 28644 ts_tablet_manager.cc:1367] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:13.842381 28641 raft_consensus.cc:348] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.842490 28641 raft_consensus.cc:374] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:13.842535 28641 raft_consensus.cc:729] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Initialized, Role: FOLLOWER
I20240501 14:09:13.842665 28641 consensus_queue.cc:260] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.842969 28641 ts_tablet_manager.cc:1398] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:13.842614 28644 raft_consensus.cc:348] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.843320 28641 tablet_bootstrap.cc:492] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312: Bootstrap starting.
I20240501 14:09:13.843327 28644 raft_consensus.cc:374] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:13.843370 28644 raft_consensus.cc:729] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Initialized, Role: FOLLOWER
I20240501 14:09:13.843469 28644 consensus_queue.cc:260] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.843700 28644 ts_tablet_manager.cc:1398] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:13.844197 28641 tablet_bootstrap.cc:654] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:13.844292 28640 tablet_bootstrap.cc:492] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170: No bootstrap required, opened a new log
I20240501 14:09:13.844375 28640 ts_tablet_manager.cc:1367] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:13.844741 28640 raft_consensus.cc:348] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.844949 28640 raft_consensus.cc:374] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:13.844996 28640 raft_consensus.cc:729] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Initialized, Role: FOLLOWER
I20240501 14:09:13.845140 28640 consensus_queue.cc:260] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:13.845443 28641 tablet_bootstrap.cc:492] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312: No bootstrap required, opened a new log
I20240501 14:09:13.845546 28641 ts_tablet_manager.cc:1367] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:13.845984 28641 raft_consensus.cc:348] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } }
I20240501 14:09:13.846117 28641 raft_consensus.cc:374] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:13.846194 28641 raft_consensus.cc:729] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Initialized, Role: FOLLOWER
I20240501 14:09:13.846330 28641 consensus_queue.cc:260] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } }
I20240501 14:09:13.846601 28641 ts_tablet_manager.cc:1398] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:13.846781 28640 ts_tablet_manager.cc:1398] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:13.898886 28471 raft_consensus.cc:482] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:13.899013 28471 raft_consensus.cc:504] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } }
I20240501 14:09:13.899350 28471 leader_election.cc:288] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:13.899824 27853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d8c8dbc45b449583f0a1808c634855" candidate_uuid: "c9cc3f1efaf44a4895d3109beabe5312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" is_pre_election: true
I20240501 14:09:13.899986 27853 raft_consensus.cc:2454] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9cc3f1efaf44a4895d3109beabe5312 in term 0.
I20240501 14:09:13.900301 27527 leader_election.cc:302] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, d1f8294a92e74484ab4a653a4ecf84e5; no voters:
I20240501 14:09:13.900519 28471 raft_consensus.cc:2781] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:13.900585 28471 raft_consensus.cc:482] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:13.900615 28471 raft_consensus.cc:3037] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:13.901504 28471 raft_consensus.cc:504] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } }
I20240501 14:09:13.901829 28471 leader_election.cc:288] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 election: Requested vote from peers d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:13.902243 27853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d8c8dbc45b449583f0a1808c634855" candidate_uuid: "c9cc3f1efaf44a4895d3109beabe5312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f8294a92e74484ab4a653a4ecf84e5"
I20240501 14:09:13.902384 27853 raft_consensus.cc:3037] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:13.903167 27853 raft_consensus.cc:2454] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9cc3f1efaf44a4895d3109beabe5312 in term 1.
I20240501 14:09:13.903470 27527 leader_election.cc:302] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, d1f8294a92e74484ab4a653a4ecf84e5; no voters:
I20240501 14:09:13.903656 28471 raft_consensus.cc:2781] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:13.903752 28471 raft_consensus.cc:686] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Becoming Leader. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Running, Role: LEADER
I20240501 14:09:13.903905 28471 consensus_queue.cc:237] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } }
I20240501 14:09:13.905494 27442 catalog_manager.cc:5484] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:14.143150 28633 raft_consensus.cc:482] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:14.143316 28633 raft_consensus.cc:504] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:14.143677 28633 leader_election.cc:288] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:14.144179 27722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbe4acb729e48bb89d5a26cb211b2eb" candidate_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec89cd1e3ba491eab76811aae4af170" is_pre_election: true
I20240501 14:09:14.144333 27722 raft_consensus.cc:2454] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f8294a92e74484ab4a653a4ecf84e5 in term 0.
I20240501 14:09:14.144634 27787 leader_election.cc:302] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d1f8294a92e74484ab4a653a4ecf84e5, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:14.144816 28633 raft_consensus.cc:2781] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:14.144891 28633 raft_consensus.cc:482] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:14.144919 28633 raft_consensus.cc:3037] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:14.145725 28633 raft_consensus.cc:504] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:14.145995 28633 leader_election.cc:288] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 election: Requested vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:14.146349 27722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbe4acb729e48bb89d5a26cb211b2eb" candidate_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
I20240501 14:09:14.146451 27722 raft_consensus.cc:3037] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:14.147130 27722 raft_consensus.cc:2454] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f8294a92e74484ab4a653a4ecf84e5 in term 1.
I20240501 14:09:14.147400 27787 leader_election.cc:302] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d1f8294a92e74484ab4a653a4ecf84e5, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:14.147555 28633 raft_consensus.cc:2781] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:14.147634 28633 raft_consensus.cc:686] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Becoming Leader. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Running, Role: LEADER
I20240501 14:09:14.147771 28633 consensus_queue.cc:237] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:14.149380 27442 catalog_manager.cc:5484] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67). New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:14.200990 28619 raft_consensus.cc:482] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:14.201121 28619 raft_consensus.cc:504] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:14.201535 28619 leader_election.cc:288] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:14.204910 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf4c3bcfc524e708092cd92d0622684" candidate_uuid: "fec89cd1e3ba491eab76811aae4af170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312" is_pre_election: true
I20240501 14:09:14.205058 27591 raft_consensus.cc:2454] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec89cd1e3ba491eab76811aae4af170 in term 0.
I20240501 14:09:14.205374 27655 leader_election.cc:302] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:14.205564 28619 raft_consensus.cc:2781] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:14.205626 28619 raft_consensus.cc:482] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:14.205662 28619 raft_consensus.cc:3037] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:14.206475 28619 raft_consensus.cc:504] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:14.206782 28619 leader_election.cc:288] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 1 election: Requested vote from peers c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:14.207139 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf4c3bcfc524e708092cd92d0622684" candidate_uuid: "fec89cd1e3ba491eab76811aae4af170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
I20240501 14:09:14.207249 27591 raft_consensus.cc:3037] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:14.208026 27591 raft_consensus.cc:2454] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec89cd1e3ba491eab76811aae4af170 in term 1.
I20240501 14:09:14.208287 27655 leader_election.cc:302] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:14.208439 28619 raft_consensus.cc:2781] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:14.208518 28619 raft_consensus.cc:686] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Becoming Leader. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Running, Role: LEADER
I20240501 14:09:14.208650 28619 consensus_queue.cc:237] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:14.210266 27442 catalog_manager.cc:5484] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec89cd1e3ba491eab76811aae4af170 (127.23.191.66). New cstate: current_term: 1 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:14.299440 28471 consensus_queue.cc:1035] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:14.390347 28653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:14.390630 28653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:14.393733 28653 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 14:09:14.470867 27591 raft_consensus.cc:1261] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:14.471422 28619 consensus_queue.cc:1035] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:14.476946 27722 raft_consensus.cc:1261] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:14.477494 28633 consensus_queue.cc:1035] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 80.8854 ms
time per row: 0.0404427 ms
I20240501 14:09:14.515616 27440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:14.521071 27570 tablet_service.cc:1467] Processing CreateTablet for tablet 085c3852d1ee47dba35aa465b3e36337 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7230a83999e24b5fb8afb43b5c8944e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:14.521093 27701 tablet_service.cc:1467] Processing CreateTablet for tablet 085c3852d1ee47dba35aa465b3e36337 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7230a83999e24b5fb8afb43b5c8944e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:14.521402 27701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 085c3852d1ee47dba35aa465b3e36337. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:14.521414 27570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 085c3852d1ee47dba35aa465b3e36337. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:14.521648 27833 tablet_service.cc:1467] Processing CreateTablet for tablet fd80131af5484a938b410e65b6ea73c7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7230a83999e24b5fb8afb43b5c8944e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:14.521788 27702 tablet_service.cc:1467] Processing CreateTablet for tablet fd80131af5484a938b410e65b6ea73c7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7230a83999e24b5fb8afb43b5c8944e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:14.521907 27833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd80131af5484a938b410e65b6ea73c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:14.521989 27702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd80131af5484a938b410e65b6ea73c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:14.523842 27832 tablet_service.cc:1467] Processing CreateTablet for tablet 322635d2e89b4bd69e1f1442b0f811e8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7230a83999e24b5fb8afb43b5c8944e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:14.523963 28701 tablet_bootstrap.cc:492] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312: Bootstrap starting.
I20240501 14:09:14.524065 27832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 322635d2e89b4bd69e1f1442b0f811e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:14.524178 28702 tablet_bootstrap.cc:492] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5: Bootstrap starting.
I20240501 14:09:14.524920 28701 tablet_bootstrap.cc:654] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:14.525002 28702 tablet_bootstrap.cc:654] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:14.525943 28700 tablet_bootstrap.cc:492] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170: Bootstrap starting.
I20240501 14:09:14.525970 27570 tablet_service.cc:1467] Processing CreateTablet for tablet 322635d2e89b4bd69e1f1442b0f811e8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7230a83999e24b5fb8afb43b5c8944e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:14.526139 28701 tablet_bootstrap.cc:492] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312: No bootstrap required, opened a new log
I20240501 14:09:14.526212 27570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 322635d2e89b4bd69e1f1442b0f811e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:14.526341 28701 ts_tablet_manager.cc:1367] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:14.526846 28700 tablet_bootstrap.cc:654] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:14.526840 28701 raft_consensus.cc:348] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:14.526949 28701 raft_consensus.cc:374] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:14.526988 28701 raft_consensus.cc:729] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Initialized, Role: FOLLOWER
I20240501 14:09:14.527156 28701 consensus_queue.cc:260] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:14.527535 28701 ts_tablet_manager.cc:1398] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:14.527904 28700 tablet_bootstrap.cc:492] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170: No bootstrap required, opened a new log
I20240501 14:09:14.527932 28701 tablet_bootstrap.cc:492] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312: Bootstrap starting.
I20240501 14:09:14.527997 28700 ts_tablet_manager.cc:1367] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:14.528429 28700 raft_consensus.cc:348] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:14.528540 28700 raft_consensus.cc:374] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:14.528576 28700 raft_consensus.cc:729] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Initialized, Role: FOLLOWER
I20240501 14:09:14.528710 28700 consensus_queue.cc:260] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:14.529011 28700 ts_tablet_manager.cc:1398] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:14.529148 28702 tablet_bootstrap.cc:492] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5: No bootstrap required, opened a new log
I20240501 14:09:14.529183 28700 tablet_bootstrap.cc:492] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170: Bootstrap starting.
I20240501 14:09:14.529157 28701 tablet_bootstrap.cc:654] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:14.529243 28702 ts_tablet_manager.cc:1367] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:14.529714 28702 raft_consensus.cc:348] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:14.529825 28702 raft_consensus.cc:374] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:14.529860 28702 raft_consensus.cc:729] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Initialized, Role: FOLLOWER
I20240501 14:09:14.529989 28700 tablet_bootstrap.cc:654] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:14.530017 28702 consensus_queue.cc:260] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:14.531278 28702 ts_tablet_manager.cc:1398] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:14.531442 28702 tablet_bootstrap.cc:492] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5: Bootstrap starting.
I20240501 14:09:14.532024 28700 tablet_bootstrap.cc:492] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170: No bootstrap required, opened a new log
I20240501 14:09:14.532104 28700 ts_tablet_manager.cc:1367] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:14.532524 28700 raft_consensus.cc:348] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:14.532634 28700 raft_consensus.cc:374] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:14.532671 28700 raft_consensus.cc:729] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Initialized, Role: FOLLOWER
I20240501 14:09:14.532691 28702 tablet_bootstrap.cc:654] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:14.532785 28700 consensus_queue.cc:260] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:14.533077 28700 ts_tablet_manager.cc:1398] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:14.533099 28701 tablet_bootstrap.cc:492] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312: No bootstrap required, opened a new log
I20240501 14:09:14.533272 28701 ts_tablet_manager.cc:1367] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:09:14.533735 28701 raft_consensus.cc:348] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:14.533844 28701 raft_consensus.cc:374] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:14.533874 28701 raft_consensus.cc:729] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Initialized, Role: FOLLOWER
I20240501 14:09:14.533999 28701 consensus_queue.cc:260] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:14.534353 28701 ts_tablet_manager.cc:1398] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:14.535337 28702 tablet_bootstrap.cc:492] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5: No bootstrap required, opened a new log
I20240501 14:09:14.535526 28702 ts_tablet_manager.cc:1367] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:09:14.535957 28702 raft_consensus.cc:348] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:14.536072 28702 raft_consensus.cc:374] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:14.536121 28702 raft_consensus.cc:729] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Initialized, Role: FOLLOWER
I20240501 14:09:14.536238 28702 consensus_queue.cc:260] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:14.536530 28702 ts_tablet_manager.cc:1398] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:14.557435 28633 raft_consensus.cc:482] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:14.557554 28633 raft_consensus.cc:504] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:14.557897 28633 leader_election.cc:288] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:14.558320 27722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd80131af5484a938b410e65b6ea73c7" candidate_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec89cd1e3ba491eab76811aae4af170" is_pre_election: true
I20240501 14:09:14.558462 27722 raft_consensus.cc:2454] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f8294a92e74484ab4a653a4ecf84e5 in term 0.
I20240501 14:09:14.558784 27787 leader_election.cc:302] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d1f8294a92e74484ab4a653a4ecf84e5, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:14.558957 28633 raft_consensus.cc:2781] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:14.559022 28633 raft_consensus.cc:482] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:14.559058 28633 raft_consensus.cc:3037] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:14.559844 28633 raft_consensus.cc:504] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:14.560106 28633 leader_election.cc:288] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 election: Requested vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:14.560478 27722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd80131af5484a938b410e65b6ea73c7" candidate_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
I20240501 14:09:14.560590 27722 raft_consensus.cc:3037] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:14.561347 27722 raft_consensus.cc:2454] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f8294a92e74484ab4a653a4ecf84e5 in term 1.
I20240501 14:09:14.561614 27787 leader_election.cc:302] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d1f8294a92e74484ab4a653a4ecf84e5, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:14.561774 28633 raft_consensus.cc:2781] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:14.561861 28633 raft_consensus.cc:686] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Becoming Leader. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Running, Role: LEADER
I20240501 14:09:14.562017 28633 consensus_queue.cc:237] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:14.563519 27440 catalog_manager.cc:5484] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67). New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:14.639072 28619 raft_consensus.cc:482] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:14.639216 28619 raft_consensus.cc:504] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:14.639595 28619 leader_election.cc:288] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:14.640096 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085c3852d1ee47dba35aa465b3e36337" candidate_uuid: "fec89cd1e3ba491eab76811aae4af170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312" is_pre_election: true
I20240501 14:09:14.640257 27591 raft_consensus.cc:2454] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec89cd1e3ba491eab76811aae4af170 in term 0.
I20240501 14:09:14.640565 27655 leader_election.cc:302] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:14.640739 28619 raft_consensus.cc:2781] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:14.640798 28619 raft_consensus.cc:482] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:14.640856 28619 raft_consensus.cc:3037] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:14.641650 28619 raft_consensus.cc:504] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:14.641943 28619 leader_election.cc:288] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 1 election: Requested vote from peers c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:14.642426 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085c3852d1ee47dba35aa465b3e36337" candidate_uuid: "fec89cd1e3ba491eab76811aae4af170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
I20240501 14:09:14.642568 27591 raft_consensus.cc:3037] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:14.643568 27591 raft_consensus.cc:2454] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec89cd1e3ba491eab76811aae4af170 in term 1.
I20240501 14:09:14.643858 27655 leader_election.cc:302] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:14.644048 28619 raft_consensus.cc:2781] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:14.644143 28619 raft_consensus.cc:686] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Becoming Leader. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Running, Role: LEADER
I20240501 14:09:14.644290 28619 consensus_queue.cc:237] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } }
I20240501 14:09:14.645918 27440 catalog_manager.cc:5484] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec89cd1e3ba491eab76811aae4af170 (127.23.191.66). New cstate: current_term: 1 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:14.907918 28633 raft_consensus.cc:482] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:14.908061 28633 raft_consensus.cc:504] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:14.908440 28633 leader_election.cc:288] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:14.908998 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322635d2e89b4bd69e1f1442b0f811e8" candidate_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312" is_pre_election: true
I20240501 14:09:14.909149 27591 raft_consensus.cc:2454] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f8294a92e74484ab4a653a4ecf84e5 in term 0.
I20240501 14:09:14.909451 27786 leader_election.cc:302] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, d1f8294a92e74484ab4a653a4ecf84e5; no voters:
I20240501 14:09:14.909622 28633 raft_consensus.cc:2781] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:14.909677 28633 raft_consensus.cc:482] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:14.909708 28633 raft_consensus.cc:3037] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:14.910482 28633 raft_consensus.cc:504] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:14.910734 28633 leader_election.cc:288] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 election: Requested vote from peers c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:14.911111 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322635d2e89b4bd69e1f1442b0f811e8" candidate_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
I20240501 14:09:14.911229 27591 raft_consensus.cc:3037] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:14.911971 27591 raft_consensus.cc:2454] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f8294a92e74484ab4a653a4ecf84e5 in term 1.
I20240501 14:09:14.912248 27786 leader_election.cc:302] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, d1f8294a92e74484ab4a653a4ecf84e5; no voters:
I20240501 14:09:14.912434 28633 raft_consensus.cc:2781] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:14.912526 28633 raft_consensus.cc:686] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Becoming Leader. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Running, Role: LEADER
I20240501 14:09:14.912669 28633 consensus_queue.cc:237] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } }
I20240501 14:09:14.914270 27440 catalog_manager.cc:5484] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67). New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:15.088689 28712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:15.089001 28712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:15.092087 28712 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 14:09:15.145005 28633 consensus_queue.cc:1035] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:15.177268 27591 raft_consensus.cc:1261] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:15.177858 28633 consensus_queue.cc:1035] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:15.178297 27591 raft_consensus.cc:1261] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:15.178925 28619 consensus_queue.cc:1035] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 85.1639 ms
time per row: 0.0425819 ms
I20240501 14:09:15.217761 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.68:43375
--local_ip_for_outbound_sockets=127.23.191.68
--tserver_master_addrs=127.23.191.126:38485
--webserver_port=37467
--webserver_interface=127.23.191.68
--builtin_ntp_servers=127.23.191.84:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:15.326759 28763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:15.326990 28763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:15.327034 28763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:15.330174 28763 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:15.330252 28763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:15.330332 28763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.68
I20240501 14:09:15.334144 28763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.68:43375
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.23.191.68
--webserver_port=37467
--tserver_master_addrs=127.23.191.126:38485
--never_fsync=true
--heap_profile_path=/tmp/kudu.28763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.68
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:15.335072 28763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:15.336227 28763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:15.344617 28768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.344980 28771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.345655 28769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.346114 28763 server_base.cc:1034] running on GCE node
I20240501 14:09:15.346540 28763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:15.347124 28763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:15.348280 28763 hybrid_clock.cc:648] HybridClock initialized: now 1714572555348266 us; error 30 us; skew 500 ppm
I20240501 14:09:15.350095 28763 webserver.cc:460] Webserver started at http://127.23.191.68:37467/ using document root <none> and password file <none>
I20240501 14:09:15.350648 28763 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:15.350725 28763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:15.354280 28763 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20240501 14:09:15.356333 28777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.357450 28763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:15.357556 28763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4c37ce2bb5d844c09969e644d463cf17"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:15.357970 28763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:15.373739 28763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:15.374379 28763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:15.374536 28763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:15.375182 28763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:15.376451 28784 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 14:09:15.379575 28763 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 14:09:15.379640 28763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240501 14:09:15.379698 28763 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 14:09:15.381162 28763 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 14:09:15.381206 28763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:15.381441 28784 tablet_bootstrap.cc:492] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17: Bootstrap starting.
I20240501 14:09:15.390918 28763 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.68:43375
I20240501 14:09:15.390995 28891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.68:43375 every 8 connection(s)
I20240501 14:09:15.391995 28763 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:09:15.393505 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 28763
I20240501 14:09:15.393895 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.69:39239
--local_ip_for_outbound_sockets=127.23.191.69
--tserver_master_addrs=127.23.191.126:38485
--webserver_port=39265
--webserver_interface=127.23.191.69
--builtin_ntp_servers=127.23.191.84:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:15.394817 28784 log.cc:824] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:15.396324 28784 tablet_bootstrap.cc:492] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:15.396631 28784 tablet_bootstrap.cc:492] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17: Bootstrap complete.
I20240501 14:09:15.397835 28784 ts_tablet_manager.cc:1367] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.009s
I20240501 14:09:15.401490 28892 heartbeater.cc:347] Connected to a master server at 127.23.191.126:38485
I20240501 14:09:15.401722 28892 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:15.401429 28784 raft_consensus.cc:348] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } }
I20240501 14:09:15.401860 28784 raft_consensus.cc:729] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c37ce2bb5d844c09969e644d463cf17, State: Initialized, Role: FOLLOWER
I20240501 14:09:15.402345 28892 heartbeater.cc:511] Master 127.23.191.126:38485 requested a full tablet report, sending...
I20240501 14:09:15.402451 28784 consensus_queue.cc:260] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } }
I20240501 14:09:15.403466 28784 ts_tablet_manager.cc:1398] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 14:09:15.404301 27443 ts_manager.cc:194] Re-registered known tserver with Master: 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68:43375)
I20240501 14:09:15.405112 27443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.68:35189
I20240501 14:09:15.406878 28892 heartbeater.cc:503] Master 127.23.191.126:38485 was elected leader, sending a full tablet report...
W20240501 14:09:15.512137 28896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:15.512383 28896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:15.512442 28896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:15.515697 28896 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:15.515767 28896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:15.515833 28896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.69
I20240501 14:09:15.519615 28896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.69:39239
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.23.191.69
--webserver_port=39265
--tserver_master_addrs=127.23.191.126:38485
--never_fsync=true
--heap_profile_path=/tmp/kudu.28896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.69
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:15.520541 28896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:15.521800 28896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:15.530400 28905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.530966 28902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.531440 28903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.532002 28896 server_base.cc:1034] running on GCE node
I20240501 14:09:15.532368 28896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:15.532936 28896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:15.534090 28896 hybrid_clock.cc:648] HybridClock initialized: now 1714572555534073 us; error 32 us; skew 500 ppm
I20240501 14:09:15.541018 28896 webserver.cc:460] Webserver started at http://127.23.191.69:39265/ using document root <none> and password file <none>
I20240501 14:09:15.541585 28896 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:15.541657 28896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:15.546295 28896 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:15.548503 28911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.549734 28896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:15.549839 28896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ae4fd219bcae42e3969310879588dc32"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:15.550290 28896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:15.570467 28896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:15.571183 28896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:15.571342 28896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:15.571902 28896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:15.572808 28896 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:15.572891 28896 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.572939 28896 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:15.572968 28896 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.581492 28896 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.69:39239
I20240501 14:09:15.581518 29024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.69:39239 every 8 connection(s)
I20240501 14:09:15.582603 28896 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:09:15.590992 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 28896
I20240501 14:09:15.591435 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.70:33907
--local_ip_for_outbound_sockets=127.23.191.70
--tserver_master_addrs=127.23.191.126:38485
--webserver_port=33593
--webserver_interface=127.23.191.70
--builtin_ntp_servers=127.23.191.84:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:15.591632 29025 heartbeater.cc:347] Connected to a master server at 127.23.191.126:38485
I20240501 14:09:15.591864 29025 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:15.592396 29025 heartbeater.cc:511] Master 127.23.191.126:38485 requested a full tablet report, sending...
I20240501 14:09:15.593518 27443 ts_manager.cc:194] Re-registered known tserver with Master: ae4fd219bcae42e3969310879588dc32 (127.23.191.69:39239)
I20240501 14:09:15.594084 27443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.69:57969
W20240501 14:09:15.642936 28893 tablet.cc:2369] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:09:15.712078 29028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:15.712301 29028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:15.712347 29028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:15.716327 29028 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:15.716428 29028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:15.716529 29028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.70
I20240501 14:09:15.721351 29028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.70:33907
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.23.191.70
--webserver_port=33593
--tserver_master_addrs=127.23.191.126:38485
--never_fsync=true
--heap_profile_path=/tmp/kudu.29028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.70
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:15.722347 29028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:15.723537 29028 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:15.731789 29033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.732115 29034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.732731 29036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.733201 29028 server_base.cc:1034] running on GCE node
I20240501 14:09:15.733582 29028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:15.734144 29028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:15.735297 29028 hybrid_clock.cc:648] HybridClock initialized: now 1714572555735285 us; error 31 us; skew 500 ppm
I20240501 14:09:15.737200 29028 webserver.cc:460] Webserver started at http://127.23.191.70:33593/ using document root <none> and password file <none>
I20240501 14:09:15.737748 29028 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:15.737841 29028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:15.741293 29028 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:15.743227 29042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.744213 29028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:15.744309 29028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3625e18e2ea14d43aa3b86b014fd89f5"
format_stamp: "Formatted at 2024-05-01 14:09:07 on dist-test-slave-c4g3"
I20240501 14:09:15.744725 29028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:15.761087 29028 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:15.761844 29028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:15.761996 29028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:15.762554 29028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:15.763689 29049 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 14:09:15.766803 29028 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 14:09:15.766865 29028 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240501 14:09:15.766919 29028 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 14:09:15.768399 29028 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 14:09:15.768445 29028 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240501 14:09:15.768692 29049 tablet_bootstrap.cc:492] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap starting.
I20240501 14:09:15.778004 29028 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.70:33907
I20240501 14:09:15.778067 29156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.70:33907 every 8 connection(s)
I20240501 14:09:15.779227 29028 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:09:15.783615 29049 log.cc:824] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:15.785765 29049 tablet_bootstrap.cc:492] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:15.786170 29049 tablet_bootstrap.cc:492] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap complete.
I20240501 14:09:15.787124 29049 ts_tablet_manager.cc:1367] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20240501 14:09:15.788333 29157 heartbeater.cc:347] Connected to a master server at 127.23.191.126:38485
I20240501 14:09:15.788547 29157 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:15.789094 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 29028
I20240501 14:09:15.789146 29157 heartbeater.cc:511] Master 127.23.191.126:38485 requested a full tablet report, sending...
I20240501 14:09:15.789527 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.71:33963
--local_ip_for_outbound_sockets=127.23.191.71
--tserver_master_addrs=127.23.191.126:38485
--webserver_port=42275
--webserver_interface=127.23.191.71
--builtin_ntp_servers=127.23.191.84:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:15.790366 29049 raft_consensus.cc:348] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } }
I20240501 14:09:15.790828 27443 ts_manager.cc:194] Re-registered known tserver with Master: 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70:33907)
I20240501 14:09:15.790982 29049 raft_consensus.cc:729] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3625e18e2ea14d43aa3b86b014fd89f5, State: Initialized, Role: FOLLOWER
I20240501 14:09:15.791462 29049 consensus_queue.cc:260] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } }
I20240501 14:09:15.791702 27443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.70:54329
I20240501 14:09:15.792340 29049 ts_tablet_manager.cc:1398] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20240501 14:09:15.793939 29157 heartbeater.cc:503] Master 127.23.191.126:38485 was elected leader, sending a full tablet report...
W20240501 14:09:15.901170 29161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:15.901412 29161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:15.901472 29161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:15.904678 29161 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:15.904767 29161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:15.904882 29161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.71
I20240501 14:09:15.908576 29161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.71:33963
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.23.191.71
--webserver_port=42275
--tserver_master_addrs=127.23.191.126:38485
--never_fsync=true
--heap_profile_path=/tmp/kudu.29161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.71
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:15.909541 29161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:15.910763 29161 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:15.919262 29168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.919584 29170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.920248 29167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.920763 29161 server_base.cc:1034] running on GCE node
I20240501 14:09:15.921180 29161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:15.921770 29161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:15.922924 29161 hybrid_clock.cc:648] HybridClock initialized: now 1714572555922906 us; error 33 us; skew 500 ppm
I20240501 14:09:15.924762 29161 webserver.cc:460] Webserver started at http://127.23.191.71:42275/ using document root <none> and password file <none>
I20240501 14:09:15.925345 29161 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:15.925415 29161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:15.928782 29161 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:09:15.930732 29176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.931982 29161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:15.932092 29161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7"
format_stamp: "Formatted at 2024-05-01 14:09:08 on dist-test-slave-c4g3"
I20240501 14:09:15.932646 29161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:15.967610 29161 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:15.968315 29161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:15.968489 29161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:15.969082 29161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:15.970075 29161 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:15.970122 29161 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.970166 29161 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:15.970201 29161 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.978417 29161 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.71:33963
I20240501 14:09:15.978457 29289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.71:33963 every 8 connection(s)
I20240501 14:09:15.979454 29161 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:09:15.987869 29290 heartbeater.cc:347] Connected to a master server at 127.23.191.126:38485
I20240501 14:09:15.988103 29290 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:15.988369 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 29161
I20240501 14:09:15.988513 24317 test_util.cc:268] Using random seed: 136588834
I20240501 14:09:15.988677 29290 heartbeater.cc:511] Master 127.23.191.126:38485 requested a full tablet report, sending...
I20240501 14:09:15.989675 27443 ts_manager.cc:194] Re-registered known tserver with Master: ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71:33963)
I20240501 14:09:15.990268 27443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.71:58547
I20240501 14:09:15.997422 24317 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 14:09:15.997659 24317 test_util.cc:268] Using random seed: 136597977
I20240501 14:09:16.007313 24317 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 14:09:16.007477 24317 test_util.cc:268] Using random seed: 136607796
I20240501 14:09:16.020237 24317 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 14:09:16.020401 24317 test_util.cc:268] Using random seed: 136620718
W20240501 14:09:16.031172 29158 tablet.cc:2369] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:16.036793 24317 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 14:09:16.037009 24317 test_util.cc:268] Using random seed: 136637325
I20240501 14:09:16.052855 24317 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240501 14:09:16.496378 27832 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:16.496461 27570 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:16.498677 28826 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:16.515646 27702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:16.526362 29091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:16.534556 29224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:16.557152 28959 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:16.596457 29025 heartbeater.cc:503] Master 127.23.191.126:38485 was elected leader, sending a full tablet report...
I20240501 14:09:16.944207 29091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:16.946563 27570 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:16.955173 27832 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:16.969537 27702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:16.969955 29224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:16.972184 28959 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:16.992451 29290 heartbeater.cc:503] Master 127.23.191.126:38485 was elected leader, sending a full tablet report...
I20240501 14:09:16.997197 28826 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:17.001271 29469 raft_consensus.cc:482] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:17.001577 29469 raft_consensus.cc:504] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } }
I20240501 14:09:17.003297 29469 leader_election.cc:288] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68:43375)
I20240501 14:09:17.014711 28846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78612cef5af34df3b33567744725af28" candidate_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c37ce2bb5d844c09969e644d463cf17" is_pre_election: true
I20240501 14:09:17.015070 28846 raft_consensus.cc:2454] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3625e18e2ea14d43aa3b86b014fd89f5 in term 1.
I20240501 14:09:17.015498 29045 leader_election.cc:302] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3625e18e2ea14d43aa3b86b014fd89f5, 4c37ce2bb5d844c09969e644d463cf17; no voters:
I20240501 14:09:17.016331 29469 raft_consensus.cc:2781] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:09:17.016422 29469 raft_consensus.cc:482] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:17.016470 29469 raft_consensus.cc:3037] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:17.018678 29469 raft_consensus.cc:504] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } }
I20240501 14:09:17.018992 29469 leader_election.cc:288] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [CANDIDATE]: Term 2 election: Requested vote from peers 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68:43375)
I20240501 14:09:17.019634 28846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78612cef5af34df3b33567744725af28" candidate_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c37ce2bb5d844c09969e644d463cf17"
I20240501 14:09:17.019783 28846 raft_consensus.cc:3037] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:17.022037 28846 raft_consensus.cc:2454] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3625e18e2ea14d43aa3b86b014fd89f5 in term 2.
I20240501 14:09:17.026791 29045 leader_election.cc:302] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3625e18e2ea14d43aa3b86b014fd89f5, 4c37ce2bb5d844c09969e644d463cf17; no voters:
I20240501 14:09:17.029924 29469 raft_consensus.cc:2781] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:17.031025 29469 raft_consensus.cc:686] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 2 LEADER]: Becoming Leader. State: Replica: 3625e18e2ea14d43aa3b86b014fd89f5, State: Running, Role: LEADER
I20240501 14:09:17.031424 29469 consensus_queue.cc:237] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } }
I20240501 14:09:17.035959 27445 catalog_manager.cc:5484] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:17.098387 27591 consensus_queue.cc:237] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } }
I20240501 14:09:17.107832 27853 raft_consensus.cc:1261] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20240501 14:09:17.108805 29375 consensus_queue.cc:1035] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 14:09:17.120527 29375 raft_consensus.cc:2932] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER ae4fd219bcae42e3969310879588dc32 (127.23.191.69) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } } }
I20240501 14:09:17.127007 27851 raft_consensus.cc:2932] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER ae4fd219bcae42e3969310879588dc32 (127.23.191.69) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } } }
I20240501 14:09:17.130156 27445 catalog_manager.cc:5484] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index -1 to 35, NON_VOTER ae4fd219bcae42e3969310879588dc32 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } } }
W20240501 14:09:17.135223 27527 consensus_peers.cc:485] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 -> Peer ae4fd219bcae42e3969310879588dc32 (127.23.191.69:39239): Couldn't send request to peer ae4fd219bcae42e3969310879588dc32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad72089b2cf54cd0b61ee8bb2c95975f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:17.175034 27851 consensus_queue.cc:237] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } }
I20240501 14:09:17.176774 27718 raft_consensus.cc:1261] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20240501 14:09:17.177939 29386 consensus_queue.cc:1035] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20240501 14:09:17.187453 29445 raft_consensus.cc:2932] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } } }
W20240501 14:09:17.189474 27789 consensus_peers.cc:485] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 -> Peer ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71:33963): Couldn't send request to peer ff8b0a555ccd48bcb11dca38d3b2f2f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a5ec8adfb714bf98dcf3279b9df2148. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:17.192016 27445 catalog_manager.cc:5484] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index -1 to 41, NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:17.193475 27719 raft_consensus.cc:2932] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } } }
I20240501 14:09:17.253731 27851 consensus_queue.cc:237] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } }
I20240501 14:09:17.256124 27591 raft_consensus.cc:1261] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 1 index: 44. (index mismatch)
I20240501 14:09:17.256812 29445 consensus_queue.cc:1035] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20240501 14:09:17.259333 29445 raft_consensus.cc:2932] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } } }
I20240501 14:09:17.261759 27590 raft_consensus.cc:2932] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } } }
I20240501 14:09:17.262398 27444 catalog_manager.cc:5484] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index -1 to 44, NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:17.274784 27788 consensus_peers.cc:485] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 -> Peer 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70:33907): Couldn't send request to peer 3625e18e2ea14d43aa3b86b014fd89f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e06d6b042b604fd19f19c44c3161beb8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:17.291049 27851 consensus_queue.cc:237] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.47, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } }
I20240501 14:09:17.293545 27720 raft_consensus.cc:1261] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 1 index: 48. (index mismatch)
I20240501 14:09:17.293923 29445 consensus_queue.cc:1035] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20240501 14:09:17.296108 29458 raft_consensus.cc:2932] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.48: config changed from index -1 to 48, NON_VOTER 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } } }
I20240501 14:09:17.300283 27445 catalog_manager.cc:5484] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index -1 to 48, NON_VOTER 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) added. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:17.302217 27720 raft_consensus.cc:2932] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Committing config change with OpId 1.48: config changed from index -1 to 48, NON_VOTER 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } } }
I20240501 14:09:17.316506 27853 consensus_queue.cc:237] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } }
I20240501 14:09:17.318835 27718 raft_consensus.cc:1261] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 1 index: 44. (index mismatch)
I20240501 14:09:17.319238 28633 consensus_queue.cc:1035] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
W20240501 14:09:17.323254 27788 consensus_peers.cc:485] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 -> Peer 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68:43375): Couldn't send request to peer 4c37ce2bb5d844c09969e644d463cf17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd80131af5484a938b410e65b6ea73c7. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:17.327809 27789 consensus_peers.cc:485] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 -> Peer ae4fd219bcae42e3969310879588dc32 (127.23.191.69:39239): Couldn't send request to peer ae4fd219bcae42e3969310879588dc32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcbe4acb729e48bb89d5a26cb211b2eb. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:17.331962 28470 raft_consensus.cc:2932] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER ae4fd219bcae42e3969310879588dc32 (127.23.191.69) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } } }
I20240501 14:09:17.334712 27718 raft_consensus.cc:2932] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER ae4fd219bcae42e3969310879588dc32 (127.23.191.69) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } } }
I20240501 14:09:17.338945 27444 catalog_manager.cc:5484] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 reported cstate change: config changed from index -1 to 44, NON_VOTER ae4fd219bcae42e3969310879588dc32 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } } }
I20240501 14:09:17.370083 27591 consensus_queue.cc:237] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } }
I20240501 14:09:17.373615 27720 raft_consensus.cc:1261] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch)
I20240501 14:09:17.374095 28749 consensus_queue.cc:1035] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 38, Time since last communication: 0.000s
I20240501 14:09:17.376622 28749 raft_consensus.cc:2932] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index -1 to 40, NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } } }
I20240501 14:09:17.380401 27445 catalog_manager.cc:5484] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index -1 to 40, NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:17.381574 27527 consensus_peers.cc:485] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 -> Peer ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71:33963): Couldn't send request to peer ff8b0a555ccd48bcb11dca38d3b2f2f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b54ea6caa4f6448790b5456a2e5676c2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:17.388650 27718 raft_consensus.cc:2932] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index -1 to 40, NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } } }
I20240501 14:09:17.399838 27852 consensus_queue.cc:237] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 1.52, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } }
I20240501 14:09:17.401149 27719 raft_consensus.cc:1261] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 52. Preceding OpId from leader: term: 1 index: 53. (index mismatch)
I20240501 14:09:17.401751 28470 consensus_queue.cc:1035] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
W20240501 14:09:17.402709 27788 consensus_peers.cc:485] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 -> Peer 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70:33907): Couldn't send request to peer 3625e18e2ea14d43aa3b86b014fd89f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe7fe0171f2d477c91661a766239b9fa. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:17.403517 28470 raft_consensus.cc:2932] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.53: config changed from index -1 to 53, NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) added. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } } }
I20240501 14:09:17.406232 27718 raft_consensus.cc:2932] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Committing config change with OpId 1.53: config changed from index -1 to 53, NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) added. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } } }
I20240501 14:09:17.408479 27444 catalog_manager.cc:5484] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index -1 to 53, NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:17.473824 27589 consensus_queue.cc:237] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } }
I20240501 14:09:17.476537 27719 raft_consensus.cc:1261] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
W20240501 14:09:17.479934 27526 consensus_peers.cc:485] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 -> Peer 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68:43375): Couldn't send request to peer 4c37ce2bb5d844c09969e644d463cf17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea2e1d9ab9a54d52b89c72c4ef442ce5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:17.480460 29375 consensus_queue.cc:1035] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20240501 14:09:17.482407 29375 raft_consensus.cc:2932] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } } }
I20240501 14:09:17.484807 27719 raft_consensus.cc:2932] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } } }
I20240501 14:09:17.485340 27445 catalog_manager.cc:5484] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index -1 to 41, NON_VOTER 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) added. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:17.508613 27591 consensus_queue.cc:237] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 1.52, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } }
W20240501 14:09:17.511059 27527 consensus_peers.cc:485] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 -> Peer ae4fd219bcae42e3969310879588dc32 (127.23.191.69:39239): Couldn't send request to peer ae4fd219bcae42e3969310879588dc32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca6f19b8716843ccaabf666651584928. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:17.513437 27853 raft_consensus.cc:1261] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 52. Preceding OpId from leader: term: 1 index: 53. (index mismatch)
I20240501 14:09:17.515195 29375 consensus_queue.cc:1035] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20240501 14:09:17.521512 28749 raft_consensus.cc:2932] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.53: config changed from index -1 to 53, NON_VOTER ae4fd219bcae42e3969310879588dc32 (127.23.191.69) added. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } } }
I20240501 14:09:17.524080 27853 raft_consensus.cc:2932] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Committing config change with OpId 1.53: config changed from index -1 to 53, NON_VOTER ae4fd219bcae42e3969310879588dc32 (127.23.191.69) added. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } } }
I20240501 14:09:17.525709 27444 catalog_manager.cc:5484] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index -1 to 53, NON_VOTER ae4fd219bcae42e3969310879588dc32 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:17.557291 27719 consensus_queue.cc:237] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.47, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } }
I20240501 14:09:17.559448 27853 raft_consensus.cc:1261] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 1 index: 48. (index mismatch)
I20240501 14:09:17.559834 29350 consensus_queue.cc:1035] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20240501 14:09:17.563169 28757 raft_consensus.cc:2932] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Committing config change with OpId 1.48: config changed from index -1 to 48, NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } } }
I20240501 14:09:17.563910 27852 raft_consensus.cc:2932] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Committing config change with OpId 1.48: config changed from index -1 to 48, NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } } }
I20240501 14:09:17.566453 27445 catalog_manager.cc:5484] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 reported cstate change: config changed from index -1 to 48, NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:17.589172 27658 consensus_peers.cc:485] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 -> Peer ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71:33963): Couldn't send request to peer ff8b0a555ccd48bcb11dca38d3b2f2f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c717dce4871e459390c508e275c52b31. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:17.600767 27722 consensus_queue.cc:237] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.62, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } }
I20240501 14:09:17.603117 27589 raft_consensus.cc:1261] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240501 14:09:17.603543 29382 consensus_queue.cc:1035] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20240501 14:09:17.604980 29350 raft_consensus.cc:2932] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Committing config change with OpId 1.63: config changed from index -1 to 63, NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } } }
I20240501 14:09:17.605711 27591 raft_consensus.cc:2932] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Committing config change with OpId 1.63: config changed from index -1 to 63, NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } } }
I20240501 14:09:17.609227 27444 catalog_manager.cc:5484] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index -1 to 63, NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } } }
W20240501 14:09:17.613466 27657 consensus_peers.cc:485] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 -> Peer 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70:33907): Couldn't send request to peer 3625e18e2ea14d43aa3b86b014fd89f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cf4c3bcfc524e708092cd92d0622684. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:17.642194 29549 ts_tablet_manager.cc:902] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5: Initiating tablet copy from peer d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:17.643035 29549 tablet_copy_client.cc:308] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:45647
I20240501 14:09:17.650702 28846 raft_consensus.cc:1261] T 78612cef5af34df3b33567744725af28 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Refusing update from remote peer 3625e18e2ea14d43aa3b86b014fd89f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:09:17.651546 29554 consensus_queue.cc:1035] T 78612cef5af34df3b33567744725af28 P 3625e18e2ea14d43aa3b86b014fd89f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:17.655143 27873 tablet_copy_service.cc:140] P d1f8294a92e74484ab4a653a4ecf84e5: Received BeginTabletCopySession request for tablet e06d6b042b604fd19f19c44c3161beb8 from peer 3625e18e2ea14d43aa3b86b014fd89f5 ({username='slave'} at 127.23.191.70:47893)
I20240501 14:09:17.655292 27873 tablet_copy_service.cc:161] P d1f8294a92e74484ab4a653a4ecf84e5: Beginning new tablet copy session on tablet e06d6b042b604fd19f19c44c3161beb8 from peer 3625e18e2ea14d43aa3b86b014fd89f5 at {username='slave'} at 127.23.191.70:47893: session id = 3625e18e2ea14d43aa3b86b014fd89f5-e06d6b042b604fd19f19c44c3161beb8
I20240501 14:09:17.656702 27873 tablet_copy_source_session.cc:205] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:17.659457 29558 ts_tablet_manager.cc:902] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32: Initiating tablet copy from peer c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:17.660647 29558 tablet_copy_client.cc:308] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:34669
I20240501 14:09:17.661828 29549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06d6b042b604fd19f19c44c3161beb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:17.667732 27611 tablet_copy_service.cc:140] P c9cc3f1efaf44a4895d3109beabe5312: Received BeginTabletCopySession request for tablet ad72089b2cf54cd0b61ee8bb2c95975f from peer ae4fd219bcae42e3969310879588dc32 ({username='slave'} at 127.23.191.69:48699)
I20240501 14:09:17.667886 27611 tablet_copy_service.cc:161] P c9cc3f1efaf44a4895d3109beabe5312: Beginning new tablet copy session on tablet ad72089b2cf54cd0b61ee8bb2c95975f from peer ae4fd219bcae42e3969310879588dc32 at {username='slave'} at 127.23.191.69:48699: session id = ae4fd219bcae42e3969310879588dc32-ad72089b2cf54cd0b61ee8bb2c95975f
I20240501 14:09:17.669317 27611 tablet_copy_source_session.cc:205] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:17.671698 27591 consensus_queue.cc:237] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } }
I20240501 14:09:17.673599 29549 tablet_copy_client.cc:783] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:17.673789 29549 tablet_copy_client.cc:647] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:17.674121 29558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad72089b2cf54cd0b61ee8bb2c95975f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:17.675910 27722 raft_consensus.cc:1261] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240501 14:09:17.676370 29527 consensus_queue.cc:1035] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240501 14:09:17.679174 29549 tablet_copy_client.cc:523] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 14:09:17.680696 27526 consensus_peers.cc:485] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 -> Peer 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68:43375): Couldn't send request to peer 4c37ce2bb5d844c09969e644d463cf17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0d87c63b9f64212a3b451bb4140979e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:17.680752 29549 tablet_bootstrap.cc:492] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap starting.
I20240501 14:09:17.681759 29375 raft_consensus.cc:2932] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index -1 to 77, NON_VOTER 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } } }
I20240501 14:09:17.683210 29558 tablet_copy_client.cc:783] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:17.683437 29558 tablet_copy_client.cc:647] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:17.683979 27445 catalog_manager.cc:5484] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index -1 to 77, NON_VOTER 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) added. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:17.684967 27718 raft_consensus.cc:2932] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index -1 to 77, NON_VOTER 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } } }
I20240501 14:09:17.701704 29558 tablet_copy_client.cc:523] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:17.704164 29558 tablet_bootstrap.cc:492] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32: Bootstrap starting.
I20240501 14:09:17.717845 29558 log.cc:824] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:17.766695 27591 consensus_queue.cc:237] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.59, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } }
W20240501 14:09:17.770423 27527 consensus_peers.cc:485] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 -> Peer ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71:33963): Couldn't send request to peer ff8b0a555ccd48bcb11dca38d3b2f2f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41d8c8dbc45b449583f0a1808c634855. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:17.770807 27852 raft_consensus.cc:1261] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240501 14:09:17.771200 29373 consensus_queue.cc:1035] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20240501 14:09:17.779791 29558 tablet_bootstrap.cc:492] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32: Bootstrap replayed 1/1 log segments. Stats: ops{read=48 overwritten=0 applied=48 ignored=0} inserts{seen=679 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:17.780196 29558 tablet_bootstrap.cc:492] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32: Bootstrap complete.
I20240501 14:09:17.781150 29558 ts_tablet_manager.cc:1367] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32: Time spent bootstrapping tablet: real 0.077s user 0.066s sys 0.004s
I20240501 14:09:17.781198 29549 tablet_bootstrap.cc:492] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=54 overwritten=0 applied=53 ignored=0} inserts{seen=738 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:17.781507 29549 tablet_bootstrap.cc:492] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap complete.
I20240501 14:09:17.781673 29549 ts_tablet_manager.cc:1367] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5: Time spent bootstrapping tablet: real 0.101s user 0.063s sys 0.002s
I20240501 14:09:17.782168 29549 raft_consensus.cc:348] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } }
I20240501 14:09:17.783306 29549 raft_consensus.cc:729] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3625e18e2ea14d43aa3b86b014fd89f5, State: Initialized, Role: LEARNER
I20240501 14:09:17.783443 29549 consensus_queue.cc:260] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 53, Last appended: 1.54, Last appended by leader: 54, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } }
I20240501 14:09:17.783735 29549 ts_tablet_manager.cc:1398] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:09:17.784294 27873 tablet_copy_service.cc:342] P d1f8294a92e74484ab4a653a4ecf84e5: Request end of tablet copy session 3625e18e2ea14d43aa3b86b014fd89f5-e06d6b042b604fd19f19c44c3161beb8 received from {username='slave'} at 127.23.191.70:47893
I20240501 14:09:17.784389 27873 tablet_copy_service.cc:433] P d1f8294a92e74484ab4a653a4ecf84e5: ending tablet copy session 3625e18e2ea14d43aa3b86b014fd89f5-e06d6b042b604fd19f19c44c3161beb8 on tablet e06d6b042b604fd19f19c44c3161beb8 with peer 3625e18e2ea14d43aa3b86b014fd89f5
I20240501 14:09:17.785674 28749 raft_consensus.cc:2932] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.60: config changed from index -1 to 60, NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) added. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } } }
I20240501 14:09:17.788609 27444 catalog_manager.cc:5484] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index -1 to 60, NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:17.788972 29558 raft_consensus.cc:348] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } }
I20240501 14:09:17.789278 29558 raft_consensus.cc:729] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ae4fd219bcae42e3969310879588dc32, State: Initialized, Role: LEARNER
I20240501 14:09:17.789645 29558 consensus_queue.cc:260] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 48, Last appended: 1.48, Last appended by leader: 48, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } }
I20240501 14:09:17.790644 29558 ts_tablet_manager.cc:1398] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240501 14:09:17.791118 29575 ts_tablet_manager.cc:902] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Initiating tablet copy from peer d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:17.792009 29575 tablet_copy_client.cc:308] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:45647
I20240501 14:09:17.795032 27611 tablet_copy_service.cc:342] P c9cc3f1efaf44a4895d3109beabe5312: Request end of tablet copy session ae4fd219bcae42e3969310879588dc32-ad72089b2cf54cd0b61ee8bb2c95975f received from {username='slave'} at 127.23.191.69:48699
I20240501 14:09:17.795123 27611 tablet_copy_service.cc:433] P c9cc3f1efaf44a4895d3109beabe5312: ending tablet copy session ae4fd219bcae42e3969310879588dc32-ad72089b2cf54cd0b61ee8bb2c95975f on tablet ad72089b2cf54cd0b61ee8bb2c95975f with peer ae4fd219bcae42e3969310879588dc32
I20240501 14:09:17.797185 29558 ts_tablet_manager.cc:902] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32: Initiating tablet copy from peer d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:17.797668 29558 tablet_copy_client.cc:308] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:45647
I20240501 14:09:17.798880 27852 raft_consensus.cc:2932] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index -1 to 60, NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) added. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } } }
I20240501 14:09:17.810338 27873 tablet_copy_service.cc:140] P d1f8294a92e74484ab4a653a4ecf84e5: Received BeginTabletCopySession request for tablet 8a5ec8adfb714bf98dcf3279b9df2148 from peer ff8b0a555ccd48bcb11dca38d3b2f2f7 ({username='slave'} at 127.23.191.71:39425)
I20240501 14:09:17.810456 27873 tablet_copy_service.cc:161] P d1f8294a92e74484ab4a653a4ecf84e5: Beginning new tablet copy session on tablet 8a5ec8adfb714bf98dcf3279b9df2148 from peer ff8b0a555ccd48bcb11dca38d3b2f2f7 at {username='slave'} at 127.23.191.71:39425: session id = ff8b0a555ccd48bcb11dca38d3b2f2f7-8a5ec8adfb714bf98dcf3279b9df2148
I20240501 14:09:17.811352 27873 tablet_copy_source_session.cc:205] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:17.816159 29575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5ec8adfb714bf98dcf3279b9df2148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:17.824175 27852 consensus_queue.cc:237] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 1.68, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } }
I20240501 14:09:17.824510 29575 tablet_copy_client.cc:783] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:17.824770 29575 tablet_copy_client.cc:647] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Starting download of 1 WAL segments...
W20240501 14:09:17.826303 27788 consensus_peers.cc:485] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 -> Peer 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70:33907): Couldn't send request to peer 3625e18e2ea14d43aa3b86b014fd89f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 322635d2e89b4bd69e1f1442b0f811e8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:17.826776 29575 tablet_copy_client.cc:523] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:17.827242 27591 raft_consensus.cc:1261] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240501 14:09:17.827644 29445 consensus_queue.cc:1035] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20240501 14:09:17.830230 29575 tablet_bootstrap.cc:492] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Bootstrap starting.
I20240501 14:09:17.830722 28633 raft_consensus.cc:2932] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.69: config changed from index -1 to 69, NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } } }
I20240501 14:09:17.833698 27444 catalog_manager.cc:5484] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index -1 to 69, NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:17.834667 27589 raft_consensus.cc:2932] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index -1 to 69, NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } } }
I20240501 14:09:17.843623 29575 log.cc:824] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:17.863469 27873 tablet_copy_service.cc:140] P d1f8294a92e74484ab4a653a4ecf84e5: Received BeginTabletCopySession request for tablet dcbe4acb729e48bb89d5a26cb211b2eb from peer ae4fd219bcae42e3969310879588dc32 ({username='slave'} at 127.23.191.69:52049)
I20240501 14:09:17.863587 27873 tablet_copy_service.cc:161] P d1f8294a92e74484ab4a653a4ecf84e5: Beginning new tablet copy session on tablet dcbe4acb729e48bb89d5a26cb211b2eb from peer ae4fd219bcae42e3969310879588dc32 at {username='slave'} at 127.23.191.69:52049: session id = ae4fd219bcae42e3969310879588dc32-dcbe4acb729e48bb89d5a26cb211b2eb
I20240501 14:09:17.864452 27873 tablet_copy_source_session.cc:205] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:17.865870 29558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbe4acb729e48bb89d5a26cb211b2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:17.868090 29558 tablet_copy_client.cc:783] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:17.868263 29558 tablet_copy_client.cc:647] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:17.871274 29558 tablet_copy_client.cc:523] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:17.873952 29558 tablet_bootstrap.cc:492] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32: Bootstrap starting.
I20240501 14:09:17.888756 29576 ts_tablet_manager.cc:902] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32: Initiating tablet copy from peer c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:17.889376 29576 tablet_copy_client.cc:308] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:34669
I20240501 14:09:17.890203 27611 tablet_copy_service.cc:140] P c9cc3f1efaf44a4895d3109beabe5312: Received BeginTabletCopySession request for tablet ca6f19b8716843ccaabf666651584928 from peer ae4fd219bcae42e3969310879588dc32 ({username='slave'} at 127.23.191.69:48699)
I20240501 14:09:17.890309 27611 tablet_copy_service.cc:161] P c9cc3f1efaf44a4895d3109beabe5312: Beginning new tablet copy session on tablet ca6f19b8716843ccaabf666651584928 from peer ae4fd219bcae42e3969310879588dc32 at {username='slave'} at 127.23.191.69:48699: session id = ae4fd219bcae42e3969310879588dc32-ca6f19b8716843ccaabf666651584928
I20240501 14:09:17.891237 27611 tablet_copy_source_session.cc:205] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:17.892563 29576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6f19b8716843ccaabf666651584928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:17.895370 29576 tablet_copy_client.cc:783] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:17.895699 29576 tablet_copy_client.cc:647] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:17.902571 29576 tablet_copy_client.cc:523] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:17.904415 29576 tablet_bootstrap.cc:492] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32: Bootstrap starting.
I20240501 14:09:17.913023 29606 ts_tablet_manager.cc:902] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17: Initiating tablet copy from peer d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:17.913893 29606 tablet_copy_client.cc:308] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:45647
I20240501 14:09:17.940629 29549 ts_tablet_manager.cc:902] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5: Initiating tablet copy from peer d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:17.941129 29549 tablet_copy_client.cc:308] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:45647
I20240501 14:09:17.942215 27873 tablet_copy_service.cc:140] P d1f8294a92e74484ab4a653a4ecf84e5: Received BeginTabletCopySession request for tablet fe7fe0171f2d477c91661a766239b9fa from peer 3625e18e2ea14d43aa3b86b014fd89f5 ({username='slave'} at 127.23.191.70:47893)
I20240501 14:09:17.942328 27873 tablet_copy_service.cc:161] P d1f8294a92e74484ab4a653a4ecf84e5: Beginning new tablet copy session on tablet fe7fe0171f2d477c91661a766239b9fa from peer 3625e18e2ea14d43aa3b86b014fd89f5 at {username='slave'} at 127.23.191.70:47893: session id = 3625e18e2ea14d43aa3b86b014fd89f5-fe7fe0171f2d477c91661a766239b9fa
I20240501 14:09:17.943239 27873 tablet_copy_source_session.cc:205] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:17.943456 27872 tablet_copy_service.cc:140] P d1f8294a92e74484ab4a653a4ecf84e5: Received BeginTabletCopySession request for tablet fd80131af5484a938b410e65b6ea73c7 from peer 4c37ce2bb5d844c09969e644d463cf17 ({username='slave'} at 127.23.191.68:37375)
I20240501 14:09:17.943537 27872 tablet_copy_service.cc:161] P d1f8294a92e74484ab4a653a4ecf84e5: Beginning new tablet copy session on tablet fd80131af5484a938b410e65b6ea73c7 from peer 4c37ce2bb5d844c09969e644d463cf17 at {username='slave'} at 127.23.191.68:37375: session id = 4c37ce2bb5d844c09969e644d463cf17-fd80131af5484a938b410e65b6ea73c7
I20240501 14:09:17.944371 27872 tablet_copy_source_session.cc:205] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:17.946382 29606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd80131af5484a938b410e65b6ea73c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:17.949002 29549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7fe0171f2d477c91661a766239b9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:17.950085 29606 tablet_copy_client.cc:783] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:17.950317 29606 tablet_copy_client.cc:647] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:17.952723 29606 tablet_copy_client.cc:523] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:17.954418 29606 tablet_bootstrap.cc:492] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17: Bootstrap starting.
I20240501 14:09:17.958042 29549 tablet_copy_client.cc:783] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:17.958226 29549 tablet_copy_client.cc:647] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:17.961732 29549 tablet_copy_client.cc:523] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:17.963253 29549 tablet_bootstrap.cc:492] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap starting.
I20240501 14:09:17.975410 29627 ts_tablet_manager.cc:902] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Initiating tablet copy from peer c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:17.975976 29627 tablet_copy_client.cc:308] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:34669
I20240501 14:09:17.982605 27611 tablet_copy_service.cc:140] P c9cc3f1efaf44a4895d3109beabe5312: Received BeginTabletCopySession request for tablet b54ea6caa4f6448790b5456a2e5676c2 from peer ff8b0a555ccd48bcb11dca38d3b2f2f7 ({username='slave'} at 127.23.191.71:58061)
I20240501 14:09:17.982728 27611 tablet_copy_service.cc:161] P c9cc3f1efaf44a4895d3109beabe5312: Beginning new tablet copy session on tablet b54ea6caa4f6448790b5456a2e5676c2 from peer ff8b0a555ccd48bcb11dca38d3b2f2f7 at {username='slave'} at 127.23.191.71:58061: session id = ff8b0a555ccd48bcb11dca38d3b2f2f7-b54ea6caa4f6448790b5456a2e5676c2
I20240501 14:09:17.983639 27611 tablet_copy_source_session.cc:205] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:17.990144 29627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54ea6caa4f6448790b5456a2e5676c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:17.996918 29627 tablet_copy_client.cc:783] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:17.997341 29627 tablet_copy_client.cc:647] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:18.014891 29627 tablet_copy_client.cc:523] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:18.016425 29627 tablet_bootstrap.cc:492] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Bootstrap starting.
I20240501 14:09:18.023770 29558 tablet_bootstrap.cc:492] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32: Bootstrap replayed 1/1 log segments. Stats: ops{read=63 overwritten=0 applied=63 ignored=0} inserts{seen=735 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:18.024367 29558 tablet_bootstrap.cc:492] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32: Bootstrap complete.
I20240501 14:09:18.024578 29558 ts_tablet_manager.cc:1367] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32: Time spent bootstrapping tablet: real 0.151s user 0.061s sys 0.008s
I20240501 14:09:18.025578 29558 raft_consensus.cc:348] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } }
I20240501 14:09:18.025839 29558 raft_consensus.cc:729] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ae4fd219bcae42e3969310879588dc32, State: Initialized, Role: LEARNER
I20240501 14:09:18.025970 29558 consensus_queue.cc:260] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 63, Last appended: 1.63, Last appended by leader: 63, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } }
I20240501 14:09:18.026242 29558 ts_tablet_manager.cc:1398] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:18.026888 27872 tablet_copy_service.cc:342] P d1f8294a92e74484ab4a653a4ecf84e5: Request end of tablet copy session ae4fd219bcae42e3969310879588dc32-dcbe4acb729e48bb89d5a26cb211b2eb received from {username='slave'} at 127.23.191.69:52049
I20240501 14:09:18.026988 27872 tablet_copy_service.cc:433] P d1f8294a92e74484ab4a653a4ecf84e5: ending tablet copy session ae4fd219bcae42e3969310879588dc32-dcbe4acb729e48bb89d5a26cb211b2eb on tablet dcbe4acb729e48bb89d5a26cb211b2eb with peer ae4fd219bcae42e3969310879588dc32
I20240501 14:09:18.038029 29639 ts_tablet_manager.cc:902] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5: Initiating tablet copy from peer fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:18.038602 29639 tablet_copy_client.cc:308] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:44265
I20240501 14:09:18.038715 29644 ts_tablet_manager.cc:902] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17: Initiating tablet copy from peer c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:18.039373 29644 tablet_copy_client.cc:308] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:34669
I20240501 14:09:18.043776 27742 tablet_copy_service.cc:140] P fec89cd1e3ba491eab76811aae4af170: Received BeginTabletCopySession request for tablet 8cf4c3bcfc524e708092cd92d0622684 from peer 3625e18e2ea14d43aa3b86b014fd89f5 ({username='slave'} at 127.23.191.70:34939)
I20240501 14:09:18.043917 27742 tablet_copy_service.cc:161] P fec89cd1e3ba491eab76811aae4af170: Beginning new tablet copy session on tablet 8cf4c3bcfc524e708092cd92d0622684 from peer 3625e18e2ea14d43aa3b86b014fd89f5 at {username='slave'} at 127.23.191.70:34939: session id = 3625e18e2ea14d43aa3b86b014fd89f5-8cf4c3bcfc524e708092cd92d0622684
I20240501 14:09:18.044229 27611 tablet_copy_service.cc:140] P c9cc3f1efaf44a4895d3109beabe5312: Received BeginTabletCopySession request for tablet ea2e1d9ab9a54d52b89c72c4ef442ce5 from peer 4c37ce2bb5d844c09969e644d463cf17 ({username='slave'} at 127.23.191.68:44227)
I20240501 14:09:18.044327 27611 tablet_copy_service.cc:161] P c9cc3f1efaf44a4895d3109beabe5312: Beginning new tablet copy session on tablet ea2e1d9ab9a54d52b89c72c4ef442ce5 from peer 4c37ce2bb5d844c09969e644d463cf17 at {username='slave'} at 127.23.191.68:44227: session id = 4c37ce2bb5d844c09969e644d463cf17-ea2e1d9ab9a54d52b89c72c4ef442ce5
I20240501 14:09:18.045238 27611 tablet_copy_source_session.cc:205] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:18.045420 27742 tablet_copy_source_session.cc:205] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:18.046020 29644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2e1d9ab9a54d52b89c72c4ef442ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:18.046588 29639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cf4c3bcfc524e708092cd92d0622684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:18.048254 29644 tablet_copy_client.cc:783] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:18.048390 29644 tablet_copy_client.cc:647] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:18.048684 29639 tablet_copy_client.cc:783] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:18.048813 29639 tablet_copy_client.cc:647] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:18.050127 29644 tablet_copy_client.cc:523] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:18.052582 29639 tablet_copy_client.cc:523] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:18.052918 29644 tablet_bootstrap.cc:492] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17: Bootstrap starting.
I20240501 14:09:18.054237 29639 tablet_bootstrap.cc:492] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap starting.
I20240501 14:09:18.117627 29575 tablet_bootstrap.cc:492] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=57 overwritten=0 applied=57 ignored=0} inserts{seen=729 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:18.118033 29575 tablet_bootstrap.cc:492] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Bootstrap complete.
I20240501 14:09:18.118403 29575 ts_tablet_manager.cc:1367] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Time spent bootstrapping tablet: real 0.288s user 0.071s sys 0.005s
I20240501 14:09:18.120813 29575 raft_consensus.cc:348] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } }
I20240501 14:09:18.121095 29575 raft_consensus.cc:729] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ff8b0a555ccd48bcb11dca38d3b2f2f7, State: Initialized, Role: LEARNER
I20240501 14:09:18.121454 29575 consensus_queue.cc:260] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 57, Last appended: 1.57, Last appended by leader: 57, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } }
I20240501 14:09:18.122368 29575 ts_tablet_manager.cc:1398] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20240501 14:09:18.136807 27872 tablet_copy_service.cc:342] P d1f8294a92e74484ab4a653a4ecf84e5: Request end of tablet copy session ff8b0a555ccd48bcb11dca38d3b2f2f7-8a5ec8adfb714bf98dcf3279b9df2148 received from {username='slave'} at 127.23.191.71:39425
I20240501 14:09:18.136940 27872 tablet_copy_service.cc:433] P d1f8294a92e74484ab4a653a4ecf84e5: ending tablet copy session ff8b0a555ccd48bcb11dca38d3b2f2f7-8a5ec8adfb714bf98dcf3279b9df2148 on tablet 8a5ec8adfb714bf98dcf3279b9df2148 with peer ff8b0a555ccd48bcb11dca38d3b2f2f7
I20240501 14:09:18.162963 29111 raft_consensus.cc:1203] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.43->[1.44-1.61] Dedup: 1.54->[1.55-1.61]
I20240501 14:09:18.171435 29662 ts_tablet_manager.cc:902] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17: Initiating tablet copy from peer c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:18.182962 29575 ts_tablet_manager.cc:902] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Initiating tablet copy from peer fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:18.183477 29575 tablet_copy_client.cc:308] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:44265
I20240501 14:09:18.186081 29458 raft_consensus.cc:1050] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5: attempting to promote NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 to VOTER
I20240501 14:09:18.186527 29458 consensus_queue.cc:237] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 1.62, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } }
I20240501 14:09:18.188062 29111 raft_consensus.cc:1261] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240501 14:09:18.188462 28470 consensus_queue.cc:1035] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 61, Time since last communication: 0.000s
I20240501 14:09:18.190181 28470 raft_consensus.cc:2932] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.63: config changed from index 44 to 63, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.192878 27444 catalog_manager.cc:5484] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index 44 to 63, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:18.194391 29111 raft_consensus.cc:2932] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Committing config change with OpId 1.63: config changed from index 44 to 63, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.196436 27852 consensus_queue.cc:237] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.64, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } }
I20240501 14:09:18.197399 29111 raft_consensus.cc:1261] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240501 14:09:18.197825 29386 consensus_queue.cc:1035] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240501 14:09:18.199110 27590 raft_consensus.cc:1261] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240501 14:09:18.201043 29386 raft_consensus.cc:2932] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.65: config changed from index 63 to 65, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.203238 27436 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e06d6b042b604fd19f19c44c3161beb8 with cas_config_opid_index 63: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:18.207108 27444 catalog_manager.cc:5484] T e06d6b042b604fd19f19c44c3161beb8 P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index 63 to 65, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:18.209240 29111 raft_consensus.cc:2932] T e06d6b042b604fd19f19c44c3161beb8 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 63 to 65, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.212983 27570 tablet_service.cc:1514] Processing DeleteTablet for tablet e06d6b042b604fd19f19c44c3161beb8 with delete_type TABLET_DATA_TOMBSTONED (TS c9cc3f1efaf44a4895d3109beabe5312 not found in new config with opid_index 65) from {username='slave'} at 127.0.0.1:58578
I20240501 14:09:18.234886 27742 tablet_copy_service.cc:140] P fec89cd1e3ba491eab76811aae4af170: Received BeginTabletCopySession request for tablet c717dce4871e459390c508e275c52b31 from peer ff8b0a555ccd48bcb11dca38d3b2f2f7 ({username='slave'} at 127.23.191.71:53777)
I20240501 14:09:18.235013 27742 tablet_copy_service.cc:161] P fec89cd1e3ba491eab76811aae4af170: Beginning new tablet copy session on tablet c717dce4871e459390c508e275c52b31 from peer ff8b0a555ccd48bcb11dca38d3b2f2f7 at {username='slave'} at 127.23.191.71:53777: session id = ff8b0a555ccd48bcb11dca38d3b2f2f7-c717dce4871e459390c508e275c52b31
I20240501 14:09:18.235946 27742 tablet_copy_source_session.cc:205] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:18.238376 29575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c717dce4871e459390c508e275c52b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:18.240614 29575 tablet_copy_client.cc:783] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:18.240779 29575 tablet_copy_client.cc:647] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:18.244355 29675 tablet_replica.cc:330] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312: stopping tablet replica
I20240501 14:09:18.244750 29675 raft_consensus.cc:2229] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:18.245687 29244 raft_consensus.cc:1203] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.40->[1.41-1.70] Dedup: 1.57->[1.58-1.70]
I20240501 14:09:18.248392 29675 pending_rounds.cc:62] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312: Trying to abort 1 pending ops.
I20240501 14:09:18.248462 29675 pending_rounds.cc:68] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312: Aborting op as it isn't in flight: id { term: 1 index: 62 } timestamp: 7022889198319984640 op_type: WRITE_OP write_request { tablet_id: "e06d6b042b604fd19f19c44c3161beb8" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\003y\343h\216\225\335\255\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""661c367a0c792e44" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022889198290563072 authz_token { token_data: "\010\267\234\311\261\006\"3\n\005slave\022*\n ab50a0e1b4c24c32aba260261f572686\020\001\030\001 \001(\001" signature: "<redacted>""\237r\366\364>\251\261\033\335d\034\n\362\034\377\027 b \2251{\362\2377\2323\233\274\000\233{\254\372\232s\305\226\357%MB\357\375\304=\233\\\374)\367\351\321b\334\360\017}\224\375\367\260\365\356" signing_key_seq_num: 0 } } request_id { client_id: "c4b429da6e2a453e9ac68b00676df075" seq_no: 200 first_incomplete_seq_no: 200 attempt_no: 0 }
I20240501 14:09:18.248792 29675 raft_consensus.cc:2258] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:18.250018 29675 ts_tablet_manager.cc:1875] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:18.251703 28979 raft_consensus.cc:1203] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.34->[1.35-1.59] Dedup: 1.48->[1.49-1.59]
I20240501 14:09:18.252070 29575 tablet_copy_client.cc:523] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:18.253428 29575 tablet_bootstrap.cc:492] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Bootstrap starting.
I20240501 14:09:18.267222 29675 ts_tablet_manager.cc:1888] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.62
I20240501 14:09:18.267421 29675 log.cc:1196] T e06d6b042b604fd19f19c44c3161beb8 P c9cc3f1efaf44a4895d3109beabe5312: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/e06d6b042b604fd19f19c44c3161beb8
I20240501 14:09:18.268165 27433 catalog_manager.cc:4830] TS c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669): tablet e06d6b042b604fd19f19c44c3161beb8 (table rebalance_test_table_0 [id=ab50a0e1b4c24c32aba260261f572686]) successfully deleted
I20240501 14:09:18.268738 29627 tablet_bootstrap.cc:492] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=60 overwritten=0 applied=60 ignored=0} inserts{seen=745 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:18.273228 29662 tablet_copy_client.cc:308] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:34669
I20240501 14:09:18.273727 27611 tablet_copy_service.cc:140] P c9cc3f1efaf44a4895d3109beabe5312: Received BeginTabletCopySession request for tablet e0d87c63b9f64212a3b451bb4140979e from peer 4c37ce2bb5d844c09969e644d463cf17 ({username='slave'} at 127.23.191.68:44227)
I20240501 14:09:18.273820 27611 tablet_copy_service.cc:161] P c9cc3f1efaf44a4895d3109beabe5312: Beginning new tablet copy session on tablet e0d87c63b9f64212a3b451bb4140979e from peer 4c37ce2bb5d844c09969e644d463cf17 at {username='slave'} at 127.23.191.68:44227: session id = 4c37ce2bb5d844c09969e644d463cf17-e0d87c63b9f64212a3b451bb4140979e
I20240501 14:09:18.274713 27611 tablet_copy_source_session.cc:205] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:18.275584 29662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d87c63b9f64212a3b451bb4140979e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:18.275707 29627 tablet_bootstrap.cc:492] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Bootstrap complete.
I20240501 14:09:18.276118 29627 ts_tablet_manager.cc:1367] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Time spent bootstrapping tablet: real 0.260s user 0.070s sys 0.000s
I20240501 14:09:18.277951 29662 tablet_copy_client.cc:783] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:18.278191 29662 tablet_copy_client.cc:647] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:18.282886 29627 raft_consensus.cc:348] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } }
I20240501 14:09:18.283139 29627 raft_consensus.cc:729] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ff8b0a555ccd48bcb11dca38d3b2f2f7, State: Initialized, Role: LEARNER
I20240501 14:09:18.283319 29627 consensus_queue.cc:260] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 60, Last appended: 1.60, Last appended by leader: 60, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } }
I20240501 14:09:18.286106 29627 ts_tablet_manager.cc:1398] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:09:18.292743 27611 tablet_copy_service.cc:342] P c9cc3f1efaf44a4895d3109beabe5312: Request end of tablet copy session ff8b0a555ccd48bcb11dca38d3b2f2f7-b54ea6caa4f6448790b5456a2e5676c2 received from {username='slave'} at 127.23.191.71:58061
I20240501 14:09:18.292876 27611 tablet_copy_service.cc:433] P c9cc3f1efaf44a4895d3109beabe5312: ending tablet copy session ff8b0a555ccd48bcb11dca38d3b2f2f7-b54ea6caa4f6448790b5456a2e5676c2 on tablet b54ea6caa4f6448790b5456a2e5676c2 with peer ff8b0a555ccd48bcb11dca38d3b2f2f7
I20240501 14:09:18.294533 29627 ts_tablet_manager.cc:902] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Initiating tablet copy from peer c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:18.303562 29627 tablet_copy_client.cc:308] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:34669
I20240501 14:09:18.304309 27611 tablet_copy_service.cc:140] P c9cc3f1efaf44a4895d3109beabe5312: Received BeginTabletCopySession request for tablet 41d8c8dbc45b449583f0a1808c634855 from peer ff8b0a555ccd48bcb11dca38d3b2f2f7 ({username='slave'} at 127.23.191.71:58061)
I20240501 14:09:18.304409 27611 tablet_copy_service.cc:161] P c9cc3f1efaf44a4895d3109beabe5312: Beginning new tablet copy session on tablet 41d8c8dbc45b449583f0a1808c634855 from peer ff8b0a555ccd48bcb11dca38d3b2f2f7 at {username='slave'} at 127.23.191.71:58061: session id = ff8b0a555ccd48bcb11dca38d3b2f2f7-41d8c8dbc45b449583f0a1808c634855
I20240501 14:09:18.305307 27611 tablet_copy_source_session.cc:205] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:18.308100 28978 raft_consensus.cc:1203] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.43->[1.44-1.71] Dedup: 1.63->[1.64-1.71]
I20240501 14:09:18.308527 29644 tablet_bootstrap.cc:492] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17: Bootstrap replayed 1/1 log segments. Stats: ops{read=60 overwritten=0 applied=60 ignored=0} inserts{seen=745 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:18.308801 29644 tablet_bootstrap.cc:492] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17: Bootstrap complete.
I20240501 14:09:18.309002 29644 ts_tablet_manager.cc:1367] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17: Time spent bootstrapping tablet: real 0.256s user 0.062s sys 0.005s
I20240501 14:09:18.309571 29644 raft_consensus.cc:348] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } }
I20240501 14:09:18.309706 29644 raft_consensus.cc:729] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4c37ce2bb5d844c09969e644d463cf17, State: Initialized, Role: LEARNER
I20240501 14:09:18.309821 29644 consensus_queue.cc:260] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 60, Last appended: 1.60, Last appended by leader: 60, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } }
I20240501 14:09:18.310155 29644 ts_tablet_manager.cc:1398] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:18.311446 29662 tablet_copy_client.cc:523] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:18.313169 29662 tablet_bootstrap.cc:492] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17: Bootstrap starting.
I20240501 14:09:18.314430 29627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d8c8dbc45b449583f0a1808c634855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:18.329304 29627 tablet_copy_client.cc:783] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:18.329562 29627 tablet_copy_client.cc:647] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:18.346524 29627 tablet_copy_client.cc:523] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:18.348745 29627 tablet_bootstrap.cc:492] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Bootstrap starting.
I20240501 14:09:18.350652 29243 raft_consensus.cc:1203] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.39->[1.40-1.66] Dedup: 1.60->[1.61-1.66]
I20240501 14:09:18.357306 27611 tablet_copy_service.cc:342] P c9cc3f1efaf44a4895d3109beabe5312: Request end of tablet copy session 4c37ce2bb5d844c09969e644d463cf17-ea2e1d9ab9a54d52b89c72c4ef442ce5 received from {username='slave'} at 127.23.191.68:44227
I20240501 14:09:18.357442 27611 tablet_copy_service.cc:433] P c9cc3f1efaf44a4895d3109beabe5312: ending tablet copy session 4c37ce2bb5d844c09969e644d463cf17-ea2e1d9ab9a54d52b89c72c4ef442ce5 on tablet ea2e1d9ab9a54d52b89c72c4ef442ce5 with peer 4c37ce2bb5d844c09969e644d463cf17
I20240501 14:09:18.418206 29606 tablet_bootstrap.cc:492] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=70 ignored=0} inserts{seen=742 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:18.418509 29606 tablet_bootstrap.cc:492] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17: Bootstrap complete.
I20240501 14:09:18.418728 29606 ts_tablet_manager.cc:1367] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17: Time spent bootstrapping tablet: real 0.464s user 0.078s sys 0.004s
I20240501 14:09:18.427927 29606 raft_consensus.cc:348] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } }
I20240501 14:09:18.428076 29606 raft_consensus.cc:729] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4c37ce2bb5d844c09969e644d463cf17, State: Initialized, Role: LEARNER
I20240501 14:09:18.428190 29606 consensus_queue.cc:260] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 70, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } }
I20240501 14:09:18.432389 29606 ts_tablet_manager.cc:1398] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240501 14:09:18.438411 27872 tablet_copy_service.cc:342] P d1f8294a92e74484ab4a653a4ecf84e5: Request end of tablet copy session 4c37ce2bb5d844c09969e644d463cf17-fd80131af5484a938b410e65b6ea73c7 received from {username='slave'} at 127.23.191.68:37375
I20240501 14:09:18.438509 27872 tablet_copy_service.cc:433] P d1f8294a92e74484ab4a653a4ecf84e5: ending tablet copy session 4c37ce2bb5d844c09969e644d463cf17-fd80131af5484a938b410e65b6ea73c7 on tablet fd80131af5484a938b410e65b6ea73c7 with peer 4c37ce2bb5d844c09969e644d463cf17
I20240501 14:09:18.442045 29549 tablet_bootstrap.cc:492] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=66 overwritten=0 applied=66 ignored=0} inserts{seen=738 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:18.442322 29549 tablet_bootstrap.cc:492] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap complete.
I20240501 14:09:18.452971 29549 ts_tablet_manager.cc:1367] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5: Time spent bootstrapping tablet: real 0.490s user 0.069s sys 0.004s
I20240501 14:09:18.453465 29549 raft_consensus.cc:348] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } }
I20240501 14:09:18.453600 29549 raft_consensus.cc:729] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3625e18e2ea14d43aa3b86b014fd89f5, State: Initialized, Role: LEARNER
I20240501 14:09:18.453713 29549 consensus_queue.cc:260] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 66, Last appended: 1.66, Last appended by leader: 66, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } }
I20240501 14:09:18.462927 29549 ts_tablet_manager.cc:1398] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:09:18.464663 27872 tablet_copy_service.cc:342] P d1f8294a92e74484ab4a653a4ecf84e5: Request end of tablet copy session 3625e18e2ea14d43aa3b86b014fd89f5-fe7fe0171f2d477c91661a766239b9fa received from {username='slave'} at 127.23.191.70:47893
I20240501 14:09:18.464752 27872 tablet_copy_service.cc:433] P d1f8294a92e74484ab4a653a4ecf84e5: ending tablet copy session 3625e18e2ea14d43aa3b86b014fd89f5-fe7fe0171f2d477c91661a766239b9fa on tablet fe7fe0171f2d477c91661a766239b9fa with peer 3625e18e2ea14d43aa3b86b014fd89f5
I20240501 14:09:18.469331 29575 tablet_bootstrap.cc:492] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=65 overwritten=0 applied=64 ignored=0} inserts{seen=736 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:18.469391 29549 ts_tablet_manager.cc:902] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5: Initiating tablet copy from peer d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:18.469710 29575 tablet_bootstrap.cc:492] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Bootstrap complete.
I20240501 14:09:18.469862 29549 tablet_copy_client.cc:308] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:45647
I20240501 14:09:18.469950 29575 ts_tablet_manager.cc:1367] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Time spent bootstrapping tablet: real 0.217s user 0.058s sys 0.011s
I20240501 14:09:18.470413 29575 raft_consensus.cc:348] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } }
I20240501 14:09:18.471096 27872 tablet_copy_service.cc:140] P d1f8294a92e74484ab4a653a4ecf84e5: Received BeginTabletCopySession request for tablet 322635d2e89b4bd69e1f1442b0f811e8 from peer 3625e18e2ea14d43aa3b86b014fd89f5 ({username='slave'} at 127.23.191.70:47893)
I20240501 14:09:18.471197 27872 tablet_copy_service.cc:161] P d1f8294a92e74484ab4a653a4ecf84e5: Beginning new tablet copy session on tablet 322635d2e89b4bd69e1f1442b0f811e8 from peer 3625e18e2ea14d43aa3b86b014fd89f5 at {username='slave'} at 127.23.191.70:47893: session id = 3625e18e2ea14d43aa3b86b014fd89f5-322635d2e89b4bd69e1f1442b0f811e8
I20240501 14:09:18.472067 27872 tablet_copy_source_session.cc:205] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:18.473014 29627 tablet_bootstrap.cc:492] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=71 overwritten=0 applied=71 ignored=0} inserts{seen=702 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:18.473320 29627 tablet_bootstrap.cc:492] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Bootstrap complete.
I20240501 14:09:18.473511 29627 ts_tablet_manager.cc:1367] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Time spent bootstrapping tablet: real 0.125s user 0.061s sys 0.009s
I20240501 14:09:18.474839 29549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 322635d2e89b4bd69e1f1442b0f811e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:18.478942 29627 raft_consensus.cc:348] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } }
I20240501 14:09:18.479156 29627 raft_consensus.cc:729] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ff8b0a555ccd48bcb11dca38d3b2f2f7, State: Initialized, Role: LEARNER
I20240501 14:09:18.479316 29627 consensus_queue.cc:260] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 1.71, Last appended by leader: 71, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } }
I20240501 14:09:18.479673 29111 raft_consensus.cc:1203] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.52->[1.53-1.72] Dedup: 1.66->[1.67-1.72]
I20240501 14:09:18.480439 29575 raft_consensus.cc:729] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ff8b0a555ccd48bcb11dca38d3b2f2f7, State: Initialized, Role: LEARNER
I20240501 14:09:18.480566 29575 consensus_queue.cc:260] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 64, Last appended: 1.65, Last appended by leader: 65, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: true } }
I20240501 14:09:18.481341 29575 ts_tablet_manager.cc:1398] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240501 14:09:18.481974 27742 tablet_copy_service.cc:342] P fec89cd1e3ba491eab76811aae4af170: Request end of tablet copy session ff8b0a555ccd48bcb11dca38d3b2f2f7-c717dce4871e459390c508e275c52b31 received from {username='slave'} at 127.23.191.71:53777
I20240501 14:09:18.482061 27742 tablet_copy_service.cc:433] P fec89cd1e3ba491eab76811aae4af170: ending tablet copy session ff8b0a555ccd48bcb11dca38d3b2f2f7-c717dce4871e459390c508e275c52b31 on tablet c717dce4871e459390c508e275c52b31 with peer ff8b0a555ccd48bcb11dca38d3b2f2f7
I20240501 14:09:18.488765 29375 raft_consensus.cc:1050] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312: attempting to promote NON_VOTER ae4fd219bcae42e3969310879588dc32 to VOTER
I20240501 14:09:18.488772 29627 ts_tablet_manager.cc:1398] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240501 14:09:18.489162 29375 consensus_queue.cc:237] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.66, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } }
I20240501 14:09:18.490518 28977 raft_consensus.cc:1261] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32 [term 1 LEARNER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240501 14:09:18.490985 29315 consensus_queue.cc:1035] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 64, Time since last communication: 0.000s
I20240501 14:09:18.494714 27852 raft_consensus.cc:1261] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240501 14:09:18.495779 29528 consensus_queue.cc:1035] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 65, Time since last communication: 0.000s
I20240501 14:09:18.496762 27611 tablet_copy_service.cc:342] P c9cc3f1efaf44a4895d3109beabe5312: Request end of tablet copy session ff8b0a555ccd48bcb11dca38d3b2f2f7-41d8c8dbc45b449583f0a1808c634855 received from {username='slave'} at 127.23.191.71:58061
I20240501 14:09:18.496855 27611 tablet_copy_service.cc:433] P c9cc3f1efaf44a4895d3109beabe5312: ending tablet copy session ff8b0a555ccd48bcb11dca38d3b2f2f7-41d8c8dbc45b449583f0a1808c634855 on tablet 41d8c8dbc45b449583f0a1808c634855 with peer ff8b0a555ccd48bcb11dca38d3b2f2f7
I20240501 14:09:18.497413 28977 raft_consensus.cc:1203] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.65->[1.66-1.67] Dedup: 1.66->[1.67-1.67]
I20240501 14:09:18.498421 29549 tablet_copy_client.cc:783] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:18.498649 29549 tablet_copy_client.cc:647] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:18.504773 29315 raft_consensus.cc:2932] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.67: config changed from index 35 to 67, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:18.507658 27444 catalog_manager.cc:5484] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index 35 to 67, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:18.507524 28978 raft_consensus.cc:2932] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32 [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index 35 to 67, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:18.513482 27591 consensus_queue.cc:237] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 1.68, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } }
I20240501 14:09:18.514535 28978 raft_consensus.cc:1261] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240501 14:09:18.515223 29373 consensus_queue.cc:1035] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 67, Time since last communication: 0.000s
I20240501 14:09:18.517181 29549 tablet_copy_client.cc:523] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:18.518599 29549 tablet_bootstrap.cc:492] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap starting.
I20240501 14:09:18.521559 29576 tablet_bootstrap.cc:492] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=69 ignored=0} inserts{seen=700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:18.521890 29576 tablet_bootstrap.cc:492] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32: Bootstrap complete.
I20240501 14:09:18.522074 29576 ts_tablet_manager.cc:1367] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32: Time spent bootstrapping tablet: real 0.618s user 0.072s sys 0.004s
I20240501 14:09:18.522517 29576 raft_consensus.cc:348] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } }
I20240501 14:09:18.522835 29576 raft_consensus.cc:729] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ae4fd219bcae42e3969310879588dc32, State: Initialized, Role: LEARNER
I20240501 14:09:18.522966 29576 consensus_queue.cc:260] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } }
I20240501 14:09:18.523264 29576 ts_tablet_manager.cc:1398] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:18.523842 27611 tablet_copy_service.cc:342] P c9cc3f1efaf44a4895d3109beabe5312: Request end of tablet copy session ae4fd219bcae42e3969310879588dc32-ca6f19b8716843ccaabf666651584928 received from {username='slave'} at 127.23.191.69:48699
I20240501 14:09:18.523922 27611 tablet_copy_service.cc:433] P c9cc3f1efaf44a4895d3109beabe5312: ending tablet copy session ae4fd219bcae42e3969310879588dc32-ca6f19b8716843ccaabf666651584928 on tablet ca6f19b8716843ccaabf666651584928 with peer ae4fd219bcae42e3969310879588dc32
I20240501 14:09:18.528654 29639 tablet_bootstrap.cc:492] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=72 overwritten=0 applied=72 ignored=0} inserts{seen=757 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:18.528987 29639 tablet_bootstrap.cc:492] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap complete.
I20240501 14:09:18.529177 29639 ts_tablet_manager.cc:1367] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5: Time spent bootstrapping tablet: real 0.475s user 0.071s sys 0.006s
I20240501 14:09:18.529655 29639 raft_consensus.cc:348] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } }
I20240501 14:09:18.529779 29639 raft_consensus.cc:729] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3625e18e2ea14d43aa3b86b014fd89f5, State: Initialized, Role: LEARNER
I20240501 14:09:18.529896 29639 consensus_queue.cc:260] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 72, Last appended: 1.72, Last appended by leader: 72, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } }
I20240501 14:09:18.530162 29639 ts_tablet_manager.cc:1398] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:18.530624 27742 tablet_copy_service.cc:342] P fec89cd1e3ba491eab76811aae4af170: Request end of tablet copy session 3625e18e2ea14d43aa3b86b014fd89f5-8cf4c3bcfc524e708092cd92d0622684 received from {username='slave'} at 127.23.191.70:34939
I20240501 14:09:18.530702 27742 tablet_copy_service.cc:433] P fec89cd1e3ba491eab76811aae4af170: ending tablet copy session 3625e18e2ea14d43aa3b86b014fd89f5-8cf4c3bcfc524e708092cd92d0622684 on tablet 8cf4c3bcfc524e708092cd92d0622684 with peer 3625e18e2ea14d43aa3b86b014fd89f5
I20240501 14:09:18.531916 28846 raft_consensus.cc:1203] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.47->[1.48-1.94] Dedup: 1.70->[1.71-1.94]
I20240501 14:09:18.536873 28470 raft_consensus.cc:1050] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5: attempting to promote NON_VOTER ae4fd219bcae42e3969310879588dc32 to VOTER
I20240501 14:09:18.537173 28470 consensus_queue.cc:237] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } }
I20240501 14:09:18.541064 27719 raft_consensus.cc:1261] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240501 14:09:18.541544 29386 consensus_queue.cc:1035] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240501 14:09:18.541800 28979 raft_consensus.cc:1261] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32 [term 1 LEARNER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240501 14:09:18.542923 28749 raft_consensus.cc:2932] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.69: config changed from index 67 to 69, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:18.543188 28470 consensus_queue.cc:1035] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240501 14:09:18.543661 29360 raft_consensus.cc:2932] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index 44 to 75, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:18.544735 27433 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ad72089b2cf54cd0b61ee8bb2c95975f with cas_config_opid_index 67: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:18.545395 27444 catalog_manager.cc:5484] T ad72089b2cf54cd0b61ee8bb2c95975f P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index 67 to 69, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:18.545864 27719 raft_consensus.cc:2932] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 44 to 75, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:18.548350 27443 catalog_manager.cc:5484] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index 44 to 75, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:18.550657 28979 raft_consensus.cc:2932] T ad72089b2cf54cd0b61ee8bb2c95975f P ae4fd219bcae42e3969310879588dc32 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index 67 to 69, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:18.555145 27832 tablet_service.cc:1514] Processing DeleteTablet for tablet ad72089b2cf54cd0b61ee8bb2c95975f with delete_type TABLET_DATA_TOMBSTONED (TS d1f8294a92e74484ab4a653a4ecf84e5 not found in new config with opid_index 69) from {username='slave'} at 127.0.0.1:42372
I20240501 14:09:18.562402 28978 raft_consensus.cc:2932] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32 [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 44 to 75, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:18.564895 27851 consensus_queue.cc:237] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } }
I20240501 14:09:18.565677 29724 tablet_replica.cc:330] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5: stopping tablet replica
I20240501 14:09:18.566008 29724 raft_consensus.cc:2229] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:18.566352 29724 pending_rounds.cc:62] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5: Trying to abort 2 pending ops.
I20240501 14:09:18.566424 29724 pending_rounds.cc:68] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5: Aborting op as it isn't in flight: id { term: 1 index: 67 } timestamp: 7022889199570788352 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ad72089b2cf54cd0b61ee8bb2c95975f" old_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } } } new_config { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } } }
I20240501 14:09:18.566655 29724 raft_consensus.cc:2894] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Aborting config change with OpId 1.67: Aborted: Op aborted
I20240501 14:09:18.566740 29724 raft_consensus.cc:2866] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:09:18.568734 28979 raft_consensus.cc:1261] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240501 14:09:18.569200 28470 consensus_queue.cc:1035] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240501 14:09:18.572079 29360 raft_consensus.cc:2932] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index 75 to 77, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:18.572613 29458 raft_consensus.cc:1050] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5: attempting to promote NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 to VOTER
I20240501 14:09:18.572883 29458 consensus_queue.cc:237] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } }
I20240501 14:09:18.574002 27436 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dcbe4acb729e48bb89d5a26cb211b2eb with cas_config_opid_index 75: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:18.574604 27445 catalog_manager.cc:5484] T dcbe4acb729e48bb89d5a26cb211b2eb P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index 75 to 77, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:18.574795 29445 raft_consensus.cc:1050] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5: attempting to promote NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 to VOTER
I20240501 14:09:18.575579 29445 consensus_queue.cc:237] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } }
I20240501 14:09:18.575947 28979 raft_consensus.cc:2932] T dcbe4acb729e48bb89d5a26cb211b2eb P ae4fd219bcae42e3969310879588dc32 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 75 to 77, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:18.577162 27702 tablet_service.cc:1514] Processing DeleteTablet for tablet dcbe4acb729e48bb89d5a26cb211b2eb with delete_type TABLET_DATA_TOMBSTONED (TS fec89cd1e3ba491eab76811aae4af170 not found in new config with opid_index 77) from {username='slave'} at 127.0.0.1:48430
I20240501 14:09:18.579066 29528 raft_consensus.cc:1050] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312: attempting to promote NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 to VOTER
I20240501 14:09:18.579376 29528 consensus_queue.cc:237] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 1.67, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } }
I20240501 14:09:18.580924 27720 raft_consensus.cc:1261] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240501 14:09:18.581056 29244 raft_consensus.cc:1261] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240501 14:09:18.581295 28749 consensus_queue.cc:1035] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 66, Time since last communication: 0.000s
I20240501 14:09:18.581553 29110 raft_consensus.cc:1261] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240501 14:09:18.581902 29242 raft_consensus.cc:1261] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240501 14:09:18.582553 29386 consensus_queue.cc:1035] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 73, Time since last communication: 0.000s
I20240501 14:09:18.584045 27718 raft_consensus.cc:1261] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240501 14:09:18.584779 28470 raft_consensus.cc:2932] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index 41 to 75, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.585125 28749 consensus_queue.cc:1035] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 66, Time since last communication: 0.000s
I20240501 14:09:18.585971 27720 raft_consensus.cc:1261] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240501 14:09:18.586582 27720 raft_consensus.cc:1203] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.66->[1.67-1.68] Dedup: 1.67->[1.68-1.68]
I20240501 14:09:18.587286 28749 raft_consensus.cc:2932] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.68: config changed from index 40 to 68, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.587393 29242 raft_consensus.cc:2932] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 41 to 75, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.587785 29445 consensus_queue.cc:1035] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240501 14:09:18.588186 29445 consensus_queue.cc:1035] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 73, Time since last communication: 0.000s
I20240501 14:09:18.590977 27445 catalog_manager.cc:5484] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index 40 to 68, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:18.591938 27449 catalog_manager.cc:5484] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 reported cstate change: config changed from index 41 to 75, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.593899 27719 raft_consensus.cc:2932] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 41 to 75, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.594623 27720 raft_consensus.cc:2932] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index 40 to 68, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.595685 29242 raft_consensus.cc:2932] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index 40 to 68, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.598179 29360 raft_consensus.cc:2932] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index 53 to 74, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.602782 27444 catalog_manager.cc:5484] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index 53 to 74, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:18.605659 28846 raft_consensus.cc:1203] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.40->[1.41-1.70] Dedup: 1.60->[1.61-1.70]
I20240501 14:09:18.607106 27719 raft_consensus.cc:2932] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 53 to 74, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.566777 29724 pending_rounds.cc:68] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5: Aborting op as it isn't in flight: id { term: 1 index: 66 } timestamp: 7022889199543648256 op_type: WRITE_OP write_request { tablet_id: "ad72089b2cf54cd0b61ee8bb2c95975f" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\330\367O\035\204P\332h\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""23f33dcdecbd21fd" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022889199419965440 authz_token { token_data: "\010\270\234\311\261\006\"3\n\005slave\022*\n 4f58d714ee014c97a0e588bae9a3e9c3\020\001\030\001 \001(\001" signature: "<redacted>""\252;\302Dq\"\227p\306t\027\333\026P\304>>\266\361\215l\374 \033\345\326\021\376\274S\212\313\002\261N\246\241\303\220\024D\353X\005\036\311\261%\307u0\373\315C\202\t\024\006\300s\334f\363\361" signing_key_seq_num: 0 } } request_id { client_id: "5fa1700e208942dda7cf28d5f43bce66" seq_no: 191 first_incomplete_seq_no: 191 attempt_no: 0 }
I20240501 14:09:18.616647 27853 consensus_queue.cc:237] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } }
I20240501 14:09:18.616794 29724 raft_consensus.cc:2258] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:18.618443 29242 raft_consensus.cc:1261] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240501 14:09:18.618929 29729 tablet_replica.cc:330] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170: stopping tablet replica
I20240501 14:09:18.618199 29724 ts_tablet_manager.cc:1875] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:18.620106 29458 consensus_queue.cc:1035] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240501 14:09:18.623741 29386 consensus_queue.cc:1035] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240501 14:09:18.624698 29445 raft_consensus.cc:2932] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index 75 to 76, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.627065 29243 raft_consensus.cc:2932] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 75 to 76, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.627900 27436 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a5ec8adfb714bf98dcf3279b9df2148 with cas_config_opid_index 75: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:18.630688 27444 catalog_manager.cc:5484] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 reported cstate change: config changed from index 75 to 76, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.633209 29729 raft_consensus.cc:2229] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:18.633761 29729 raft_consensus.cc:2258] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:18.634534 27702 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a5ec8adfb714bf98dcf3279b9df2148 with delete_type TABLET_DATA_TOMBSTONED (TS fec89cd1e3ba491eab76811aae4af170 not found in new config with opid_index 76) from {username='slave'} at 127.0.0.1:48430
I20240501 14:09:18.638164 29110 raft_consensus.cc:2932] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 53 to 74, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.639705 29386 raft_consensus.cc:1050] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5: attempting to promote NON_VOTER 4c37ce2bb5d844c09969e644d463cf17 to VOTER
I20240501 14:09:18.640034 29386 consensus_queue.cc:237] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } }
I20240501 14:09:18.642915 27720 raft_consensus.cc:1261] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 14:09:18.643741 29360 consensus_queue.cc:1035] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240501 14:09:18.644989 29729 ts_tablet_manager.cc:1875] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:18.652078 29386 raft_consensus.cc:2932] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.96: config changed from index 48 to 96, 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:18.655123 27720 raft_consensus.cc:2932] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 48 to 96, 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:18.656371 29724 ts_tablet_manager.cc:1888] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.67
I20240501 14:09:18.656522 29724 log.cc:1196] T ad72089b2cf54cd0b61ee8bb2c95975f P d1f8294a92e74484ab4a653a4ecf84e5: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/ad72089b2cf54cd0b61ee8bb2c95975f
I20240501 14:09:18.657235 27436 catalog_manager.cc:4830] TS d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647): tablet ad72089b2cf54cd0b61ee8bb2c95975f (table rebalance_test_table_1 [id=4f58d714ee014c97a0e588bae9a3e9c3]) successfully deleted
I20240501 14:09:18.657846 27444 catalog_manager.cc:5484] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 reported cstate change: config changed from index 48 to 96, 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:18.658210 29729 ts_tablet_manager.cc:1888] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.76
I20240501 14:09:18.658344 29729 log.cc:1196] T dcbe4acb729e48bb89d5a26cb211b2eb P fec89cd1e3ba491eab76811aae4af170: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/dcbe4acb729e48bb89d5a26cb211b2eb
I20240501 14:09:18.661401 29729 tablet_replica.cc:330] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170: stopping tablet replica
I20240501 14:09:18.661504 27434 catalog_manager.cc:4830] TS fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265): tablet dcbe4acb729e48bb89d5a26cb211b2eb (table rebalance_test_table_3 [id=78f3b25b930c48d7bc71a12bc14a9aaa]) successfully deleted
I20240501 14:09:18.661573 29729 raft_consensus.cc:2229] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:18.662114 29729 raft_consensus.cc:2258] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:18.662832 29729 ts_tablet_manager.cc:1875] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:18.664882 29729 ts_tablet_manager.cc:1888] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.75
I20240501 14:09:18.665019 29729 log.cc:1196] T 8a5ec8adfb714bf98dcf3279b9df2148 P fec89cd1e3ba491eab76811aae4af170: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/8a5ec8adfb714bf98dcf3279b9df2148
I20240501 14:09:18.666139 27434 catalog_manager.cc:4830] TS fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265): tablet 8a5ec8adfb714bf98dcf3279b9df2148 (table rebalance_test_table_2 [id=8615b34ff7ef42beb6766dea8a59f98a]) successfully deleted
I20240501 14:09:18.667681 29662 tablet_bootstrap.cc:492] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17: Bootstrap replayed 1/1 log segments. Stats: ops{read=88 overwritten=0 applied=88 ignored=0} inserts{seen=719 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:18.667929 29662 tablet_bootstrap.cc:492] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17: Bootstrap complete.
I20240501 14:09:18.668197 29662 ts_tablet_manager.cc:1367] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17: Time spent bootstrapping tablet: real 0.355s user 0.083s sys 0.005s
I20240501 14:09:18.668717 29662 raft_consensus.cc:348] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } }
I20240501 14:09:18.668934 29662 raft_consensus.cc:729] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4c37ce2bb5d844c09969e644d463cf17, State: Initialized, Role: LEARNER
I20240501 14:09:18.669054 29662 consensus_queue.cc:260] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 88, Last appended: 1.88, Last appended by leader: 88, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: true } }
I20240501 14:09:18.670106 28845 raft_consensus.cc:1261] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 LEARNER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 14:09:18.671883 29662 ts_tablet_manager.cc:1398] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:18.673738 29242 raft_consensus.cc:1203] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.59->[1.60-1.77] Dedup: 1.71->[1.72-1.77]
I20240501 14:09:18.674029 27611 tablet_copy_service.cc:342] P c9cc3f1efaf44a4895d3109beabe5312: Request end of tablet copy session 4c37ce2bb5d844c09969e644d463cf17-e0d87c63b9f64212a3b451bb4140979e received from {username='slave'} at 127.23.191.68:44227
I20240501 14:09:18.674108 27611 tablet_copy_service.cc:433] P c9cc3f1efaf44a4895d3109beabe5312: ending tablet copy session 4c37ce2bb5d844c09969e644d463cf17-e0d87c63b9f64212a3b451bb4140979e on tablet e0d87c63b9f64212a3b451bb4140979e with peer 4c37ce2bb5d844c09969e644d463cf17
I20240501 14:09:18.682220 27853 consensus_queue.cc:237] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } }
I20240501 14:09:18.684597 28470 consensus_queue.cc:1035] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240501 14:09:18.688709 29110 raft_consensus.cc:1261] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240501 14:09:18.691233 29315 raft_consensus.cc:1050] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312: attempting to promote NON_VOTER 4c37ce2bb5d844c09969e644d463cf17 to VOTER
I20240501 14:09:18.691538 29315 consensus_queue.cc:237] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } }
I20240501 14:09:18.692869 28846 raft_consensus.cc:1261] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 LEARNER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240501 14:09:18.693720 29373 consensus_queue.cc:1035] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 70, Time since last communication: 0.000s
I20240501 14:09:18.694191 27720 raft_consensus.cc:1261] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240501 14:09:18.694655 29373 consensus_queue.cc:1035] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240501 14:09:18.695448 28846 raft_consensus.cc:2932] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 48 to 96, 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:18.699414 29528 raft_consensus.cc:2932] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index 41 to 72, 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:18.700273 27719 raft_consensus.cc:2932] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 41 to 72, 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:18.701011 28845 raft_consensus.cc:2932] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 41 to 72, 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:18.702490 27445 catalog_manager.cc:5484] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index 41 to 72, 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:18.707814 29386 consensus_queue.cc:1035] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 77, Time since last communication: 0.000s
I20240501 14:09:18.722044 28749 raft_consensus.cc:1050] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312: attempting to promote NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 to VOTER
I20240501 14:09:18.722412 28749 consensus_queue.cc:237] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } }
I20240501 14:09:18.726061 29243 raft_consensus.cc:1261] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240501 14:09:18.726616 29528 consensus_queue.cc:1035] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20240501 14:09:18.730340 29373 raft_consensus.cc:2932] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index 60 to 79, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.733455 29243 raft_consensus.cc:2932] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index 60 to 79, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.733564 27445 catalog_manager.cc:5484] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index 60 to 79, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:18.738682 27591 consensus_queue.cc:237] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } }
I20240501 14:09:18.739948 29242 raft_consensus.cc:1261] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240501 14:09:18.740731 28749 consensus_queue.cc:1035] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240501 14:09:18.742686 28749 raft_consensus.cc:2932] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index 79 to 80, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.743016 29243 raft_consensus.cc:2932] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 79 to 80, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.744302 27433 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 41d8c8dbc45b449583f0a1808c634855 with cas_config_opid_index 79: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:18.745131 27445 catalog_manager.cc:5484] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index 79 to 80, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:18.746106 29549 tablet_bootstrap.cc:492] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=91 overwritten=0 applied=91 ignored=0} inserts{seen=722 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:18.746384 29549 tablet_bootstrap.cc:492] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5: Bootstrap complete.
I20240501 14:09:18.746556 29549 ts_tablet_manager.cc:1367] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5: Time spent bootstrapping tablet: real 0.228s user 0.079s sys 0.007s
I20240501 14:09:18.746980 29549 raft_consensus.cc:348] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } }
I20240501 14:09:18.747102 29549 raft_consensus.cc:729] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3625e18e2ea14d43aa3b86b014fd89f5, State: Initialized, Role: LEARNER
I20240501 14:09:18.747215 29549 consensus_queue.cc:260] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 91, Last appended: 1.91, Last appended by leader: 91, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: true } }
I20240501 14:09:18.747474 29549 ts_tablet_manager.cc:1398] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:18.747596 27832 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d8c8dbc45b449583f0a1808c634855 with delete_type TABLET_DATA_TOMBSTONED (TS d1f8294a92e74484ab4a653a4ecf84e5 not found in new config with opid_index 80) from {username='slave'} at 127.0.0.1:42372
I20240501 14:09:18.747848 29724 tablet_replica.cc:330] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5: stopping tablet replica
I20240501 14:09:18.747994 29724 raft_consensus.cc:2229] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:18.748226 29724 raft_consensus.cc:2258] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:18.748299 29445 raft_consensus.cc:2932] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index 74 to 80, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.748685 27872 tablet_copy_service.cc:342] P d1f8294a92e74484ab4a653a4ecf84e5: Request end of tablet copy session 3625e18e2ea14d43aa3b86b014fd89f5-322635d2e89b4bd69e1f1442b0f811e8 received from {username='slave'} at 127.23.191.70:47893
I20240501 14:09:18.748770 27872 tablet_copy_service.cc:433] P d1f8294a92e74484ab4a653a4ecf84e5: ending tablet copy session 3625e18e2ea14d43aa3b86b014fd89f5-322635d2e89b4bd69e1f1442b0f811e8 on tablet 322635d2e89b4bd69e1f1442b0f811e8 with peer 3625e18e2ea14d43aa3b86b014fd89f5
I20240501 14:09:18.749842 29724 ts_tablet_manager.cc:1875] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:18.751065 27436 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe7fe0171f2d477c91661a766239b9fa with cas_config_opid_index 74: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:18.751883 29724 ts_tablet_manager.cc:1888] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.77
I20240501 14:09:18.751940 29724 log.cc:1196] T 41d8c8dbc45b449583f0a1808c634855 P d1f8294a92e74484ab4a653a4ecf84e5: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/41d8c8dbc45b449583f0a1808c634855
I20240501 14:09:18.752471 27436 catalog_manager.cc:4830] TS d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647): tablet 41d8c8dbc45b449583f0a1808c634855 (table rebalance_test_table_3 [id=78f3b25b930c48d7bc71a12bc14a9aaa]) successfully deleted
I20240501 14:09:18.753868 29110 raft_consensus.cc:2932] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 74 to 80, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.756337 27444 catalog_manager.cc:5484] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 reported cstate change: config changed from index 74 to 80, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.760854 27702 tablet_service.cc:1514] Processing DeleteTablet for tablet fe7fe0171f2d477c91661a766239b9fa with delete_type TABLET_DATA_TOMBSTONED (TS fec89cd1e3ba491eab76811aae4af170 not found in new config with opid_index 80) from {username='slave'} at 127.0.0.1:48430
I20240501 14:09:18.761094 29729 tablet_replica.cc:330] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170: stopping tablet replica
I20240501 14:09:18.761265 29729 raft_consensus.cc:2229] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:18.762629 29729 pending_rounds.cc:62] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170: Trying to abort 1 pending ops.
I20240501 14:09:18.762709 29729 pending_rounds.cc:68] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170: Aborting op as it isn't in flight: id { term: 1 index: 79 } timestamp: 7022889200356429824 op_type: WRITE_OP write_request { tablet_id: "fe7fe0171f2d477c91661a766239b9fa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000+\203\264x@\317ym\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""fb43879b9cf820ac" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022889200302129152 authz_token { token_data: "\010\270\234\311\261\006\"3\n\005slave\022*\n 4f58d714ee014c97a0e588bae9a3e9c3\020\001\030\001 \001(\001" signature: "<redacted>""\252;\302Dq\"\227p\306t\027\333\026P\304>>\266\361\215l\374 \033\345\326\021\376\274S\212\313\002\261N\246\241\303\220\024D\353X\005\036\311\261%\307u0\373\315C\202\t\024\006\300s\334f\363\361" signing_key_seq_num: 0 } } request_id { client_id: "5fa1700e208942dda7cf28d5f43bce66" seq_no: 208 first_incomplete_seq_no: 208 attempt_no: 0 }
I20240501 14:09:18.763054 29729 raft_consensus.cc:2258] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:18.765185 29729 ts_tablet_manager.cc:1875] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:18.768644 29729 ts_tablet_manager.cc:1888] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.79
I20240501 14:09:18.768740 29729 log.cc:1196] T fe7fe0171f2d477c91661a766239b9fa P fec89cd1e3ba491eab76811aae4af170: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/fe7fe0171f2d477c91661a766239b9fa
I20240501 14:09:18.770407 27434 catalog_manager.cc:4830] TS fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265): tablet fe7fe0171f2d477c91661a766239b9fa (table rebalance_test_table_1 [id=4f58d714ee014c97a0e588bae9a3e9c3]) successfully deleted
I20240501 14:09:18.802492 29242 raft_consensus.cc:1203] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.47->[1.48-1.78] Dedup: 1.65->[1.66-1.78]
I20240501 14:09:18.803318 27853 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fd80131af5484a938b410e65b6ea73c7"
dest_uuid: "d1f8294a92e74484ab4a653a4ecf84e5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44354
I20240501 14:09:18.803505 27853 raft_consensus.cc:595] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:18.809526 29458 raft_consensus.cc:979] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5: : Instructing follower fec89cd1e3ba491eab76811aae4af170 to start an election
I20240501 14:09:18.809752 29445 raft_consensus.cc:1067] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Signalling peer fec89cd1e3ba491eab76811aae4af170 to start an election
I20240501 14:09:18.810421 27718 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fd80131af5484a938b410e65b6ea73c7"
dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
from {username='slave'} at 127.23.191.67:40923
I20240501 14:09:18.810580 27718 raft_consensus.cc:482] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:18.810639 27718 raft_consensus.cc:3037] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:18.811693 27718 raft_consensus.cc:504] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } }
I20240501 14:09:18.812140 27718 leader_election.cc:288] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 2 election: Requested vote from peers d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647), 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68:43375)
I20240501 14:09:18.817006 27853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd80131af5484a938b410e65b6ea73c7" candidate_uuid: "fec89cd1e3ba491eab76811aae4af170" candidate_term: 2 candidate_status { last_received { term: 1 index: 101 } } ignore_live_leader: true dest_uuid: "d1f8294a92e74484ab4a653a4ecf84e5"
I20240501 14:09:18.817116 27853 raft_consensus.cc:3032] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:18.817148 27853 raft_consensus.cc:729] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Running, Role: LEADER
I20240501 14:09:18.817287 27853 consensus_queue.cc:260] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.101, Last appended by leader: 101, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } }
I20240501 14:09:18.817445 27853 raft_consensus.cc:3037] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:18.818384 27853 raft_consensus.cc:2454] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec89cd1e3ba491eab76811aae4af170 in term 2.
W20240501 14:09:18.819214 27811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44228: Illegal state: replica d1f8294a92e74484ab4a653a4ecf84e5 is not leader of this config: current role FOLLOWER
I20240501 14:09:18.821229 27658 leader_election.cc:302] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f8294a92e74484ab4a653a4ecf84e5, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:18.821954 28757 raft_consensus.cc:2781] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:18.822052 28757 raft_consensus.cc:686] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 2 LEADER]: Becoming Leader. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Running, Role: LEADER
I20240501 14:09:18.822167 28757 consensus_queue.cc:237] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.101, Last appended by leader: 101, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } }
I20240501 14:09:18.828058 28845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd80131af5484a938b410e65b6ea73c7" candidate_uuid: "fec89cd1e3ba491eab76811aae4af170" candidate_term: 2 candidate_status { last_received { term: 1 index: 101 } } ignore_live_leader: true dest_uuid: "4c37ce2bb5d844c09969e644d463cf17"
I20240501 14:09:18.828155 28846 raft_consensus.cc:3037] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:18.828604 27853 raft_consensus.cc:1261] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 2 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 2 index: 103. (index mismatch)
I20240501 14:09:18.829087 28757 consensus_queue.cc:1035] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240501 14:09:18.829231 28846 raft_consensus.cc:1261] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 2 index: 103. (index mismatch)
I20240501 14:09:18.830771 27445 catalog_manager.cc:5484] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 reported cstate change: term changed from 1 to 2, leader changed from d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) to fec89cd1e3ba491eab76811aae4af170 (127.23.191.66). New cstate: current_term: 2 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:18.834017 28757 consensus_queue.cc:1035] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240501 14:09:18.834740 28757 raft_consensus.cc:1050] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170: attempting to promote NON_VOTER ff8b0a555ccd48bcb11dca38d3b2f2f7 to VOTER
I20240501 14:09:18.835134 28757 consensus_queue.cc:237] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } }
I20240501 14:09:18.836820 29241 raft_consensus.cc:1261] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 LEARNER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240501 14:09:18.841171 29382 consensus_queue.cc:1035] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240501 14:09:18.843698 27853 raft_consensus.cc:1261] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240501 14:09:18.846225 28757 consensus_queue.cc:1035] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240501 14:09:18.846621 29350 raft_consensus.cc:2932] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index 48 to 80, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.847410 27719 consensus_queue.cc:237] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 2.103, Last appended by leader: 101, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } }
I20240501 14:09:18.849552 28846 raft_consensus.cc:1261] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 2 index: 103. Preceding OpId from leader: term: 2 index: 104. (index mismatch)
I20240501 14:09:18.849613 29244 raft_consensus.cc:2932] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 48 to 80, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.852118 29543 consensus_queue.cc:1035] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240501 14:09:18.854862 29543 raft_consensus.cc:2932] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 2 LEADER]: Committing config change with OpId 2.104: config changed from index 96 to 104, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:18.858052 28846 raft_consensus.cc:2932] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Committing config change with OpId 2.104: config changed from index 96 to 104, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:18.861534 27449 catalog_manager.cc:5484] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7 reported cstate change: config changed from index 48 to 80, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.862169 27445 catalog_manager.cc:5484] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 reported cstate change: config changed from index 96 to 104, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New cstate: current_term: 2 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:18.864519 27434 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fd80131af5484a938b410e65b6ea73c7 with cas_config_opid_index 96: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:18.864599 27853 raft_consensus.cc:2932] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 48 to 80, ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.864934 27832 tablet_service.cc:1514] Processing DeleteTablet for tablet fd80131af5484a938b410e65b6ea73c7 with delete_type TABLET_DATA_TOMBSTONED (TS d1f8294a92e74484ab4a653a4ecf84e5 not found in new config with opid_index 104) from {username='slave'} at 127.0.0.1:42372
I20240501 14:09:18.865100 29724 tablet_replica.cc:330] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5: stopping tablet replica
I20240501 14:09:18.865198 29724 raft_consensus.cc:2229] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:18.865288 29724 raft_consensus.cc:2258] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:18.866005 29724 ts_tablet_manager.cc:1875] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:18.868126 29724 ts_tablet_manager.cc:1888] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.103
I20240501 14:09:18.868188 29724 log.cc:1196] T fd80131af5484a938b410e65b6ea73c7 P d1f8294a92e74484ab4a653a4ecf84e5: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/fd80131af5484a938b410e65b6ea73c7
I20240501 14:09:18.868688 27436 catalog_manager.cc:4830] TS d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647): tablet fd80131af5484a938b410e65b6ea73c7 (table rebalance_test_table_4 [id=7230a83999e24b5fb8afb43b5c8944e4]) successfully deleted
I20240501 14:09:18.871140 29110 raft_consensus.cc:1203] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.68->[1.69-1.102] Dedup: 1.91->[1.92-1.102]
I20240501 14:09:18.882659 27718 consensus_queue.cc:237] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } }
I20240501 14:09:18.883936 29241 raft_consensus.cc:1261] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240501 14:09:18.884732 29543 consensus_queue.cc:1035] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240501 14:09:18.889335 29350 raft_consensus.cc:2932] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index 80 to 81, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.890327 29244 raft_consensus.cc:2932] T c717dce4871e459390c508e275c52b31 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index 80 to 81, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:18.891423 27434 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c717dce4871e459390c508e275c52b31 with cas_config_opid_index 80: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:18.892158 27444 catalog_manager.cc:5484] T c717dce4871e459390c508e275c52b31 P fec89cd1e3ba491eab76811aae4af170 reported cstate change: config changed from index 80 to 81, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:18.895515 27832 tablet_service.cc:1514] Processing DeleteTablet for tablet c717dce4871e459390c508e275c52b31 with delete_type TABLET_DATA_TOMBSTONED (TS d1f8294a92e74484ab4a653a4ecf84e5 not found in new config with opid_index 81) from {username='slave'} at 127.0.0.1:42372
I20240501 14:09:18.895936 29724 tablet_replica.cc:330] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5: stopping tablet replica
I20240501 14:09:18.896044 29724 raft_consensus.cc:2229] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:18.896320 29724 raft_consensus.cc:2258] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:18.898236 27590 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b54ea6caa4f6448790b5456a2e5676c2"
dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46448
I20240501 14:09:18.898396 27590 raft_consensus.cc:595] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:18.902012 29724 ts_tablet_manager.cc:1875] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:18.904215 29724 ts_tablet_manager.cc:1888] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.80
I20240501 14:09:18.904284 29724 log.cc:1196] T c717dce4871e459390c508e275c52b31 P d1f8294a92e74484ab4a653a4ecf84e5: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/c717dce4871e459390c508e275c52b31
I20240501 14:09:18.904762 27436 catalog_manager.cc:4830] TS d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647): tablet c717dce4871e459390c508e275c52b31 (table rebalance_test_table_0 [id=ab50a0e1b4c24c32aba260261f572686]) successfully deleted
I20240501 14:09:18.905712 29360 raft_consensus.cc:1050] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5: attempting to promote NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 to VOTER
I20240501 14:09:18.906020 29360 consensus_queue.cc:237] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } }
I20240501 14:09:18.907193 27591 raft_consensus.cc:1261] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240501 14:09:18.909204 29110 raft_consensus.cc:1261] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240501 14:09:18.910238 29386 consensus_queue.cc:1035] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240501 14:09:18.910511 29386 consensus_queue.cc:1035] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240501 14:09:18.912954 29458 raft_consensus.cc:2932] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index 69 to 104, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.915575 27444 catalog_manager.cc:5484] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index 69 to 104, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:18.917574 27590 raft_consensus.cc:2932] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 69 to 104, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.917685 29110 raft_consensus.cc:2932] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 69 to 104, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.921175 28978 raft_consensus.cc:1203] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.52->[1.53-1.88] Dedup: 1.70->[1.71-1.88]
I20240501 14:09:18.938848 27853 consensus_queue.cc:237] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } }
I20240501 14:09:18.942025 29111 raft_consensus.cc:1261] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240501 14:09:18.944211 29445 consensus_queue.cc:1035] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240501 14:09:18.954118 29528 raft_consensus.cc:1050] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312: attempting to promote NON_VOTER ae4fd219bcae42e3969310879588dc32 to VOTER
I20240501 14:09:18.954448 29528 consensus_queue.cc:237] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 1.90, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } }
I20240501 14:09:18.956081 28978 raft_consensus.cc:1261] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32 [term 1 LEARNER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 89. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240501 14:09:18.956657 29315 consensus_queue.cc:1035] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240501 14:09:18.958629 27852 raft_consensus.cc:1261] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240501 14:09:18.958618 29445 raft_consensus.cc:2932] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index 104 to 105, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.959031 29111 raft_consensus.cc:2932] T 322635d2e89b4bd69e1f1442b0f811e8 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 104 to 105, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:18.960306 27436 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 322635d2e89b4bd69e1f1442b0f811e8 with cas_config_opid_index 104: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:18.961671 29375 consensus_queue.cc:1035] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20240501 14:09:18.962319 27444 catalog_manager.cc:5484] T 322635d2e89b4bd69e1f1442b0f811e8 P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index 104 to 105, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:18.966317 27570 tablet_service.cc:1514] Processing DeleteTablet for tablet 322635d2e89b4bd69e1f1442b0f811e8 with delete_type TABLET_DATA_TOMBSTONED (TS c9cc3f1efaf44a4895d3109beabe5312 not found in new config with opid_index 105) from {username='slave'} at 127.0.0.1:58578
I20240501 14:09:18.968227 29527 raft_consensus.cc:2932] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.91: config changed from index 53 to 91, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:18.968974 29789 tablet_replica.cc:330] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312: stopping tablet replica
I20240501 14:09:18.969146 29789 raft_consensus.cc:2229] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:18.969311 29789 raft_consensus.cc:2258] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:18.970316 29789 ts_tablet_manager.cc:1875] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:18.970609 27449 catalog_manager.cc:5484] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index 53 to 91, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:18.973206 29789 ts_tablet_manager.cc:1888] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.104
I20240501 14:09:18.973327 29789 log.cc:1196] T 322635d2e89b4bd69e1f1442b0f811e8 P c9cc3f1efaf44a4895d3109beabe5312: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/322635d2e89b4bd69e1f1442b0f811e8
I20240501 14:09:18.975078 27853 raft_consensus.cc:2932] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 53 to 91, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:18.975729 27433 catalog_manager.cc:4830] TS c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669): tablet 322635d2e89b4bd69e1f1442b0f811e8 (table rebalance_test_table_4 [id=7230a83999e24b5fb8afb43b5c8944e4]) successfully deleted
I20240501 14:09:18.977900 28979 raft_consensus.cc:1203] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.89->[1.90-1.91] Dedup: 1.90->[1.91-1.91]
W20240501 14:09:18.985200 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
W20240501 14:09:18.993120 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
I20240501 14:09:18.993625 28979 raft_consensus.cc:2932] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 53 to 91, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:18.994562 27591 consensus_queue.cc:237] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 1.91, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } }
W20240501 14:09:19.000680 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
I20240501 14:09:19.003775 28978 raft_consensus.cc:1261] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 92. (index mismatch)
I20240501 14:09:19.006167 29528 consensus_queue.cc:1035] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
W20240501 14:09:19.007856 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
I20240501 14:09:19.010260 29375 raft_consensus.cc:2932] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.92: config changed from index 91 to 92, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:19.012058 27433 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ca6f19b8716843ccaabf666651584928 with cas_config_opid_index 91: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:19.012991 27449 catalog_manager.cc:5484] T ca6f19b8716843ccaabf666651584928 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index 91 to 92, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:19.015568 27832 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6f19b8716843ccaabf666651584928 with delete_type TABLET_DATA_TOMBSTONED (TS d1f8294a92e74484ab4a653a4ecf84e5 not found in new config with opid_index 92) from {username='slave'} at 127.0.0.1:42372
I20240501 14:09:19.015758 29724 tablet_replica.cc:330] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5: stopping tablet replica
I20240501 14:09:19.015882 29724 raft_consensus.cc:2229] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:19.016037 29724 raft_consensus.cc:2258] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:19.017676 29724 ts_tablet_manager.cc:1875] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:09:19.019016 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
I20240501 14:09:19.019879 29724 ts_tablet_manager.cc:1888] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.91
I20240501 14:09:19.019939 29724 log.cc:1196] T ca6f19b8716843ccaabf666651584928 P d1f8294a92e74484ab4a653a4ecf84e5: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/ca6f19b8716843ccaabf666651584928
I20240501 14:09:19.019939 28978 raft_consensus.cc:2932] T ca6f19b8716843ccaabf666651584928 P ae4fd219bcae42e3969310879588dc32 [term 1 FOLLOWER]: Committing config change with OpId 1.92: config changed from index 91 to 92, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:19.021525 27436 catalog_manager.cc:4830] TS d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647): tablet ca6f19b8716843ccaabf666651584928 (table rebalance_test_table_2 [id=8615b34ff7ef42beb6766dea8a59f98a]) successfully deleted
W20240501 14:09:19.029973 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
W20240501 14:09:19.042476 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
W20240501 14:09:19.055397 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
W20240501 14:09:19.067611 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
W20240501 14:09:19.079674 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
I20240501 14:09:19.085948 27591 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea2e1d9ab9a54d52b89c72c4ef442ce5"
dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46478
I20240501 14:09:19.086076 27591 raft_consensus.cc:595] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:09:19.093560 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
I20240501 14:09:19.097373 29110 raft_consensus.cc:1203] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.62->[1.63-1.91] Dedup: 1.72->[1.73-1.91]
W20240501 14:09:19.105939 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
W20240501 14:09:19.106048 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
W20240501 14:09:19.110627 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
W20240501 14:09:19.116479 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
W20240501 14:09:19.124428 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
I20240501 14:09:19.127950 29543 raft_consensus.cc:1050] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170: attempting to promote NON_VOTER 3625e18e2ea14d43aa3b86b014fd89f5 to VOTER
I20240501 14:09:19.128283 29543 consensus_queue.cc:237] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.92, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } }
I20240501 14:09:19.129524 27591 raft_consensus.cc:1261] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240501 14:09:19.130103 29543 consensus_queue.cc:1035] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240501 14:09:19.130396 29110 raft_consensus.cc:1261] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 LEARNER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240501 14:09:19.131063 29350 consensus_queue.cc:1035] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
W20240501 14:09:19.131588 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.132071 29382 raft_consensus.cc:2932] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Committing config change with OpId 1.93: config changed from index 63 to 93, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:19.132496 27591 raft_consensus.cc:2932] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 63 to 93, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:19.134768 29110 raft_consensus.cc:2932] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 63 to 93, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:19.135344 27449 catalog_manager.cc:5484] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index 63 to 93, 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
W20240501 14:09:19.139294 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.141230 27719 consensus_queue.cc:237] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 1.94, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } }
I20240501 14:09:19.142292 29110 raft_consensus.cc:1261] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 14:09:19.143128 29350 consensus_queue.cc:1035] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
W20240501 14:09:19.143883 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
I20240501 14:09:19.144798 28757 raft_consensus.cc:2932] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Committing config change with OpId 1.95: config changed from index 93 to 95, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:19.146981 29110 raft_consensus.cc:2932] T 8cf4c3bcfc524e708092cd92d0622684 P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index 93 to 95, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } } }
I20240501 14:09:19.147640 27434 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8cf4c3bcfc524e708092cd92d0622684 with cas_config_opid_index 93: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:19.148633 27445 catalog_manager.cc:5484] T 8cf4c3bcfc524e708092cd92d0622684 P fec89cd1e3ba491eab76811aae4af170 reported cstate change: config changed from index 93 to 95, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New cstate: current_term: 1 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:09:19.151237 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.151494 27570 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cf4c3bcfc524e708092cd92d0622684 with delete_type TABLET_DATA_TOMBSTONED (TS c9cc3f1efaf44a4895d3109beabe5312 not found in new config with opid_index 95) from {username='slave'} at 127.0.0.1:58578
I20240501 14:09:19.151772 29789 tablet_replica.cc:330] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312: stopping tablet replica
I20240501 14:09:19.151911 29789 raft_consensus.cc:2229] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:19.152009 29789 pending_rounds.cc:62] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312: Trying to abort 1 pending ops.
I20240501 14:09:19.152045 29789 pending_rounds.cc:68] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312: Aborting op as it isn't in flight: id { term: 1 index: 94 } timestamp: 7022889202206392320 op_type: WRITE_OP write_request { tablet_id: "8cf4c3bcfc524e708092cd92d0622684" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\0001\347\242\003\0162-(\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""267017bf924e6247" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022889202179166208 authz_token { token_data: "\010\270\234\311\261\006\"3\n\005slave\022*\n 78f3b25b930c48d7bc71a12bc14a9aaa\020\001\030\001 \001(\001" signature: "<redacted>"">:F\221M\177\372\033Kl\375\347k\3038\243\244\271\nHdYm\300\356\351[\311\334>\200\211\241\233p*-iv`,\255\003y\363\177[L\337\201\021\226\2748O\274\345G\266\224\221\330\"N" signing_key_seq_num: 0 } } request_id { client_id: "b177c75a47b94cd8b2f90192684370d0" seq_no: 265 first_incomplete_seq_no: 265 attempt_no: 0 }
I20240501 14:09:19.152312 29789 raft_consensus.cc:2258] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:19.153220 29789 ts_tablet_manager.cc:1875] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:19.155081 29789 ts_tablet_manager.cc:1888] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.94
I20240501 14:09:19.155146 29789 log.cc:1196] T 8cf4c3bcfc524e708092cd92d0622684 P c9cc3f1efaf44a4895d3109beabe5312: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/8cf4c3bcfc524e708092cd92d0622684
I20240501 14:09:19.155884 27433 catalog_manager.cc:4830] TS c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669): tablet 8cf4c3bcfc524e708092cd92d0622684 (table rebalance_test_table_3 [id=78f3b25b930c48d7bc71a12bc14a9aaa]) successfully deleted
W20240501 14:09:19.161275 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
W20240501 14:09:19.163188 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.170082 28846 raft_consensus.cc:1203] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.76->[1.77-1.122] Dedup: 1.88->[1.89-1.122]
W20240501 14:09:19.171878 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
W20240501 14:09:19.182016 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
W20240501 14:09:19.183315 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
W20240501 14:09:19.195498 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
W20240501 14:09:19.202744 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
I20240501 14:09:19.206614 29315 raft_consensus.cc:1050] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312: attempting to promote NON_VOTER 4c37ce2bb5d844c09969e644d463cf17 to VOTER
I20240501 14:09:19.206933 29315 consensus_queue.cc:237] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 1.124, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } }
W20240501 14:09:19.207759 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.208570 27718 raft_consensus.cc:1261] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240501 14:09:19.208659 28845 raft_consensus.cc:1261] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17 [term 1 LEARNER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240501 14:09:19.208949 29527 consensus_queue.cc:1035] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240501 14:09:19.209967 29527 consensus_queue.cc:1035] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 123, Time since last communication: 0.000s
I20240501 14:09:19.212105 28845 raft_consensus.cc:1203] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.123->[1.124-1.125] Dedup: 1.124->[1.125-1.125]
I20240501 14:09:19.213471 29527 raft_consensus.cc:2932] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.125: config changed from index 77 to 125, 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:19.213903 28845 raft_consensus.cc:2932] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 77 to 125, 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:19.217087 27445 catalog_manager.cc:5484] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index 77 to 125, 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:19.217808 27718 raft_consensus.cc:2932] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 77 to 125, 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
W20240501 14:09:19.221640 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
I20240501 14:09:19.222687 27590 consensus_queue.cc:237] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 1.125, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } }
W20240501 14:09:19.223309 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.227402 28846 raft_consensus.cc:1261] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 126. (index mismatch)
I20240501 14:09:19.227739 29315 consensus_queue.cc:1035] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20240501 14:09:19.227932 27590 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b54ea6caa4f6448790b5456a2e5676c2"
dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46510
I20240501 14:09:19.228021 27590 raft_consensus.cc:595] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:19.230707 29315 raft_consensus.cc:2932] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.126: config changed from index 125 to 126, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:19.232982 28846 raft_consensus.cc:2932] T e0d87c63b9f64212a3b451bb4140979e P 4c37ce2bb5d844c09969e644d463cf17 [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index 125 to 126, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:19.234198 27433 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e0d87c63b9f64212a3b451bb4140979e with cas_config_opid_index 125: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:19.235003 27444 catalog_manager.cc:5484] T e0d87c63b9f64212a3b451bb4140979e P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index 125 to 126, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:19.237367 27702 tablet_service.cc:1514] Processing DeleteTablet for tablet e0d87c63b9f64212a3b451bb4140979e with delete_type TABLET_DATA_TOMBSTONED (TS fec89cd1e3ba491eab76811aae4af170 not found in new config with opid_index 126) from {username='slave'} at 127.0.0.1:48430
I20240501 14:09:19.237566 29729 tablet_replica.cc:330] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170: stopping tablet replica
I20240501 14:09:19.237691 29729 raft_consensus.cc:2229] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:19.237813 29729 raft_consensus.cc:2258] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:09:19.238283 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.238430 29729 ts_tablet_manager.cc:1875] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:19.240592 29729 ts_tablet_manager.cc:1888] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.125
I20240501 14:09:19.240667 29729 log.cc:1196] T e0d87c63b9f64212a3b451bb4140979e P fec89cd1e3ba491eab76811aae4af170: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/e0d87c63b9f64212a3b451bb4140979e
I20240501 14:09:19.241256 27434 catalog_manager.cc:4830] TS fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265): tablet e0d87c63b9f64212a3b451bb4140979e (table rebalance_test_table_0 [id=ab50a0e1b4c24c32aba260261f572686]) successfully deleted
W20240501 14:09:19.245599 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
W20240501 14:09:19.253572 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.262965 27590 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea2e1d9ab9a54d52b89c72c4ef442ce5"
dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46536
I20240501 14:09:19.263108 27590 raft_consensus.cc:595] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:09:19.267864 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Service unavailable: leader transfer in progress
W20240501 14:09:19.271095 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.274911 29315 raft_consensus.cc:979] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312: : Instructing follower fec89cd1e3ba491eab76811aae4af170 to start an election
I20240501 14:09:19.275010 29315 raft_consensus.cc:1067] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Signalling peer fec89cd1e3ba491eab76811aae4af170 to start an election
I20240501 14:09:19.275687 27718 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b54ea6caa4f6448790b5456a2e5676c2"
dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
from {username='slave'} at 127.23.191.65:42661
I20240501 14:09:19.275810 27718 raft_consensus.cc:482] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:19.275867 27718 raft_consensus.cc:3037] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:19.276923 27718 raft_consensus.cc:504] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } }
I20240501 14:09:19.277323 27718 leader_election.cc:288] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 2 election: Requested vote from peers c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669), ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71:33963)
I20240501 14:09:19.277630 27590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54ea6caa4f6448790b5456a2e5676c2" candidate_uuid: "fec89cd1e3ba491eab76811aae4af170" candidate_term: 2 candidate_status { last_received { term: 1 index: 78 } } ignore_live_leader: true dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
I20240501 14:09:19.277740 27590 raft_consensus.cc:3032] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:19.277781 27590 raft_consensus.cc:729] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Running, Role: LEADER
I20240501 14:09:19.277779 29242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54ea6caa4f6448790b5456a2e5676c2" candidate_uuid: "fec89cd1e3ba491eab76811aae4af170" candidate_term: 2 candidate_status { last_received { term: 1 index: 78 } } ignore_live_leader: true dest_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7"
I20240501 14:09:19.277938 29242 raft_consensus.cc:3037] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:19.277925 27590 consensus_queue.cc:260] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 78, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } }
I20240501 14:09:19.278095 27590 raft_consensus.cc:3037] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:19.279014 29242 raft_consensus.cc:2454] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec89cd1e3ba491eab76811aae4af170 in term 2.
I20240501 14:09:19.279083 27590 raft_consensus.cc:2454] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec89cd1e3ba491eab76811aae4af170 in term 2.
I20240501 14:09:19.279481 27658 leader_election.cc:302] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fec89cd1e3ba491eab76811aae4af170, ff8b0a555ccd48bcb11dca38d3b2f2f7; no voters:
I20240501 14:09:19.279628 29543 raft_consensus.cc:2781] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:19.279685 29543 raft_consensus.cc:686] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 2 LEADER]: Becoming Leader. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Running, Role: LEADER
I20240501 14:09:19.279786 29543 consensus_queue.cc:237] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 78, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } }
I20240501 14:09:19.281575 27444 catalog_manager.cc:5484] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 reported cstate change: term changed from 1 to 2, leader changed from c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) to fec89cd1e3ba491eab76811aae4af170 (127.23.191.66). New cstate: current_term: 2 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:19.289737 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
W20240501 14:09:19.293282 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46314: Illegal state: replica c9cc3f1efaf44a4895d3109beabe5312 is not leader of this config: current role FOLLOWER
W20240501 14:09:19.313300 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
W20240501 14:09:19.335553 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.347894 29242 raft_consensus.cc:1261] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 2 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 2 index: 80. (index mismatch)
I20240501 14:09:19.348811 29350 consensus_queue.cc:1035] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240501 14:09:19.349272 27590 raft_consensus.cc:1261] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 2 index: 80. (index mismatch)
I20240501 14:09:19.350129 29382 consensus_queue.cc:1035] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240501 14:09:19.355113 29764 mvcc.cc:205] Tried to move back new op lower bound from 7022889203084353536 to 7022889202811092992. Current Snapshot: MvccSnapshot[applied={T|T < 7022889203084353536}]
I20240501 14:09:19.355376 29308 mvcc.cc:205] Tried to move back new op lower bound from 7022889203084353536 to 7022889202811092992. Current Snapshot: MvccSnapshot[applied={T|T < 7022889203084353536}]
I20240501 14:09:19.358958 27718 consensus_queue.cc:237] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 2.80, Last appended by leader: 78, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } }
W20240501 14:09:19.360136 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.363670 29242 raft_consensus.cc:1261] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 2 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 2 index: 80. Preceding OpId from leader: term: 2 index: 81. (index mismatch)
I20240501 14:09:19.364285 27590 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea2e1d9ab9a54d52b89c72c4ef442ce5"
dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46558
I20240501 14:09:19.364409 27590 raft_consensus.cc:595] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:19.365521 29543 consensus_queue.cc:1035] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240501 14:09:19.368203 29886 raft_consensus.cc:2932] T b54ea6caa4f6448790b5456a2e5676c2 P fec89cd1e3ba491eab76811aae4af170 [term 2 LEADER]: Committing config change with OpId 2.81: config changed from index 68 to 81, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:19.369763 29242 raft_consensus.cc:2932] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 2 FOLLOWER]: Committing config change with OpId 2.81: config changed from index 68 to 81, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:19.373766 27444 catalog_manager.cc:5484] T b54ea6caa4f6448790b5456a2e5676c2 P ff8b0a555ccd48bcb11dca38d3b2f2f7 reported cstate change: config changed from index 68 to 81, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New cstate: current_term: 2 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } }
I20240501 14:09:19.375826 27434 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b54ea6caa4f6448790b5456a2e5676c2 with cas_config_opid_index 68: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:19.377269 27570 tablet_service.cc:1514] Processing DeleteTablet for tablet b54ea6caa4f6448790b5456a2e5676c2 with delete_type TABLET_DATA_TOMBSTONED (TS c9cc3f1efaf44a4895d3109beabe5312 not found in new config with opid_index 81) from {username='slave'} at 127.0.0.1:58578
I20240501 14:09:19.377447 29789 tablet_replica.cc:330] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312: stopping tablet replica
I20240501 14:09:19.377570 29789 raft_consensus.cc:2229] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:19.377679 29789 raft_consensus.cc:2258] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:19.378252 29789 ts_tablet_manager.cc:1875] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:19.380421 29789 ts_tablet_manager.cc:1888] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.80
I20240501 14:09:19.380501 29789 log.cc:1196] T b54ea6caa4f6448790b5456a2e5676c2 P c9cc3f1efaf44a4895d3109beabe5312: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/b54ea6caa4f6448790b5456a2e5676c2
W20240501 14:09:19.381783 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.381809 27433 catalog_manager.cc:4830] TS c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669): tablet b54ea6caa4f6448790b5456a2e5676c2 (table rebalance_test_table_1 [id=4f58d714ee014c97a0e588bae9a3e9c3]) successfully deleted
W20240501 14:09:19.405251 27549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
W20240501 14:09:19.428346 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
W20240501 14:09:19.453598 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
W20240501 14:09:19.482802 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.501945 27590 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea2e1d9ab9a54d52b89c72c4ef442ce5"
dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46582
I20240501 14:09:19.502094 27590 raft_consensus.cc:595] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:09:19.511718 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
W20240501 14:09:19.539570 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
W20240501 14:09:19.567505 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.596873 29375 raft_consensus.cc:979] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312: : Instructing follower fec89cd1e3ba491eab76811aae4af170 to start an election
I20240501 14:09:19.597115 29375 raft_consensus.cc:1067] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Signalling peer fec89cd1e3ba491eab76811aae4af170 to start an election
I20240501 14:09:19.597733 27718 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea2e1d9ab9a54d52b89c72c4ef442ce5"
dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
from {username='slave'} at 127.23.191.65:42661
I20240501 14:09:19.597877 27718 raft_consensus.cc:482] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:19.597935 27718 raft_consensus.cc:3037] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:19.599036 27718 raft_consensus.cc:504] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } }
I20240501 14:09:19.599392 27718 leader_election.cc:288] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 2 election: Requested vote from peers c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669), 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68:43375)
W20240501 14:09:19.599610 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Service unavailable: leader transfer in progress
I20240501 14:09:19.600478 28846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2e1d9ab9a54d52b89c72c4ef442ce5" candidate_uuid: "fec89cd1e3ba491eab76811aae4af170" candidate_term: 2 candidate_status { last_received { term: 1 index: 82 } } ignore_live_leader: true dest_uuid: "4c37ce2bb5d844c09969e644d463cf17"
I20240501 14:09:19.600607 28846 raft_consensus.cc:3037] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:19.600419 27590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2e1d9ab9a54d52b89c72c4ef442ce5" candidate_uuid: "fec89cd1e3ba491eab76811aae4af170" candidate_term: 2 candidate_status { last_received { term: 1 index: 82 } } ignore_live_leader: true dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
I20240501 14:09:19.601151 27590 raft_consensus.cc:3032] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:19.601197 27590 raft_consensus.cc:729] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Running, Role: LEADER
I20240501 14:09:19.601318 27590 consensus_queue.cc:260] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 82, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } }
I20240501 14:09:19.601444 27590 raft_consensus.cc:3037] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:19.601680 28846 raft_consensus.cc:2454] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec89cd1e3ba491eab76811aae4af170 in term 2.
I20240501 14:09:19.602404 27590 raft_consensus.cc:2454] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec89cd1e3ba491eab76811aae4af170 in term 2.
I20240501 14:09:19.602747 27655 leader_election.cc:302] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:19.602922 29543 raft_consensus.cc:2781] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:19.603020 29543 raft_consensus.cc:686] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 2 LEADER]: Becoming Leader. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Running, Role: LEADER
I20240501 14:09:19.603145 29543 consensus_queue.cc:237] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 82, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } }
I20240501 14:09:19.604938 27445 catalog_manager.cc:5484] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 reported cstate change: term changed from 1 to 2, leader changed from c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) to fec89cd1e3ba491eab76811aae4af170 (127.23.191.66). New cstate: current_term: 2 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:19.632645 27551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46304: Illegal state: replica c9cc3f1efaf44a4895d3109beabe5312 is not leader of this config: current role FOLLOWER
I20240501 14:09:19.671133 28845 raft_consensus.cc:1261] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 2 index: 84. (index mismatch)
I20240501 14:09:19.671614 28757 consensus_queue.cc:1035] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20240501 14:09:19.672791 27590 raft_consensus.cc:1261] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 2 index: 84. (index mismatch)
I20240501 14:09:19.675489 29737 mvcc.cc:205] Tried to move back new op lower bound from 7022889204404224000 to 7022889204135874560. Current Snapshot: MvccSnapshot[applied={T|T < 7022889204404224000 or (T in {7022889204404224000})}]
I20240501 14:09:19.677707 27718 consensus_queue.cc:237] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 2.84, Last appended by leader: 82, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } }
I20240501 14:09:19.685545 28846 raft_consensus.cc:1261] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 2 index: 84. Preceding OpId from leader: term: 2 index: 86. (index mismatch)
I20240501 14:09:19.685911 28757 consensus_queue.cc:1035] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20240501 14:09:19.690057 29382 raft_consensus.cc:2932] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 [term 2 LEADER]: Committing config change with OpId 2.85: config changed from index 72 to 85, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:19.691715 27434 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ea2e1d9ab9a54d52b89c72c4ef442ce5 with cas_config_opid_index 72: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:19.692945 27445 catalog_manager.cc:5484] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P fec89cd1e3ba491eab76811aae4af170 reported cstate change: config changed from index 72 to 85, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New cstate: current_term: 2 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:19.697072 27570 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2e1d9ab9a54d52b89c72c4ef442ce5 with delete_type TABLET_DATA_TOMBSTONED (TS c9cc3f1efaf44a4895d3109beabe5312 not found in new config with opid_index 85) from {username='slave'} at 127.0.0.1:58578
I20240501 14:09:19.697299 29789 tablet_replica.cc:330] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312: stopping tablet replica
I20240501 14:09:19.697424 29789 raft_consensus.cc:2229] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:19.697512 29789 raft_consensus.cc:2258] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:19.698535 29789 ts_tablet_manager.cc:1875] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:19.700604 29789 ts_tablet_manager.cc:1888] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.82
I20240501 14:09:19.700677 29789 log.cc:1196] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P c9cc3f1efaf44a4895d3109beabe5312: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/ea2e1d9ab9a54d52b89c72c4ef442ce5
I20240501 14:09:19.701145 27433 catalog_manager.cc:4830] TS c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669): tablet ea2e1d9ab9a54d52b89c72c4ef442ce5 (table rebalance_test_table_2 [id=8615b34ff7ef42beb6766dea8a59f98a]) successfully deleted
I20240501 14:09:19.702472 28845 raft_consensus.cc:2932] T ea2e1d9ab9a54d52b89c72c4ef442ce5 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Committing config change with OpId 2.85: config changed from index 72 to 85, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } }
I20240501 14:09:19.858282 29091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:19.860127 28826 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:19.864431 27570 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:19.867352 28959 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:19.878103 27832 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:19.911636 27702 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:19.944980 29224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:20.002750 27718 consensus_queue.cc:237] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 150, Committed index: 150, Last appended: 2.150, Last appended by leader: 101, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } }
I20240501 14:09:20.004194 28846 raft_consensus.cc:1261] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 2 index: 150. Preceding OpId from leader: term: 2 index: 151. (index mismatch)
I20240501 14:09:20.005575 28757 consensus_queue.cc:1035] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 151, Last known committed idx: 150, Time since last communication: 0.000s
I20240501 14:09:20.007404 28757 raft_consensus.cc:2932] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 2 LEADER]: Committing config change with OpId 2.151: config changed from index 104 to 151, NON_VOTER ae4fd219bcae42e3969310879588dc32 (127.23.191.69) added. New config: { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } } }
I20240501 14:09:20.010799 27449 catalog_manager.cc:5484] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 reported cstate change: config changed from index 104 to 151, NON_VOTER ae4fd219bcae42e3969310879588dc32 (127.23.191.69) added. New cstate: current_term: 2 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:20.013799 28846 raft_consensus.cc:2932] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Committing config change with OpId 2.151: config changed from index 104 to 151, NON_VOTER ae4fd219bcae42e3969310879588dc32 (127.23.191.69) added. New config: { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } } }
W20240501 14:09:20.030805 27658 consensus_peers.cc:485] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 -> Peer ae4fd219bcae42e3969310879588dc32 (127.23.191.69:39239): Couldn't send request to peer ae4fd219bcae42e3969310879588dc32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd80131af5484a938b410e65b6ea73c7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:20.522799 29983 ts_tablet_manager.cc:902] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32: Initiating tablet copy from peer fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:20.524461 29983 tablet_copy_client.cc:308] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:44265
I20240501 14:09:20.530205 27742 tablet_copy_service.cc:140] P fec89cd1e3ba491eab76811aae4af170: Received BeginTabletCopySession request for tablet fd80131af5484a938b410e65b6ea73c7 from peer ae4fd219bcae42e3969310879588dc32 ({username='slave'} at 127.23.191.69:48565)
I20240501 14:09:20.530321 27742 tablet_copy_service.cc:161] P fec89cd1e3ba491eab76811aae4af170: Beginning new tablet copy session on tablet fd80131af5484a938b410e65b6ea73c7 from peer ae4fd219bcae42e3969310879588dc32 at {username='slave'} at 127.23.191.69:48565: session id = ae4fd219bcae42e3969310879588dc32-fd80131af5484a938b410e65b6ea73c7
I20240501 14:09:20.531252 27742 tablet_copy_source_session.cc:205] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:20.532074 29983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd80131af5484a938b410e65b6ea73c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.537511 29983 tablet_copy_client.cc:783] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:20.537787 29983 tablet_copy_client.cc:647] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:20.540057 29983 tablet_copy_client.cc:523] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:20.542719 29983 tablet_bootstrap.cc:492] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32: Bootstrap starting.
I20240501 14:09:20.743767 29983 tablet_bootstrap.cc:492] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32: Bootstrap replayed 1/1 log segments. Stats: ops{read=176 overwritten=0 applied=176 ignored=0} inserts{seen=844 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:20.744143 29983 tablet_bootstrap.cc:492] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32: Bootstrap complete.
I20240501 14:09:20.744361 29983 ts_tablet_manager.cc:1367] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32: Time spent bootstrapping tablet: real 0.202s user 0.139s sys 0.007s
I20240501 14:09:20.745103 29983 raft_consensus.cc:348] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } }
I20240501 14:09:20.745246 29983 raft_consensus.cc:729] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ae4fd219bcae42e3969310879588dc32, State: Initialized, Role: LEARNER
I20240501 14:09:20.745349 29983 consensus_queue.cc:260] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 176, Last appended: 2.176, Last appended by leader: 176, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: true } }
I20240501 14:09:20.745735 29983 ts_tablet_manager.cc:1398] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:09:20.747707 27742 tablet_copy_service.cc:342] P fec89cd1e3ba491eab76811aae4af170: Request end of tablet copy session ae4fd219bcae42e3969310879588dc32-fd80131af5484a938b410e65b6ea73c7 received from {username='slave'} at 127.23.191.69:48565
I20240501 14:09:20.747795 27742 tablet_copy_service.cc:433] P fec89cd1e3ba491eab76811aae4af170: ending tablet copy session ae4fd219bcae42e3969310879588dc32-fd80131af5484a938b410e65b6ea73c7 on tablet fd80131af5484a938b410e65b6ea73c7 with peer ae4fd219bcae42e3969310879588dc32
I20240501 14:09:20.975685 28979 raft_consensus.cc:1203] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.150->[2.151-2.193] Dedup: 2.176->[2.177-2.193]
I20240501 14:09:21.353268 29886 raft_consensus.cc:1050] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170: attempting to promote NON_VOTER ae4fd219bcae42e3969310879588dc32 to VOTER
I20240501 14:09:21.353615 29886 consensus_queue.cc:237] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 194, Committed index: 194, Last appended: 2.194, Last appended by leader: 101, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } }
I20240501 14:09:21.354908 28845 raft_consensus.cc:1261] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 2 index: 194. Preceding OpId from leader: term: 2 index: 195. (index mismatch)
I20240501 14:09:21.355301 30010 consensus_queue.cc:1035] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 194, Time since last communication: 0.000s
I20240501 14:09:21.357910 29350 raft_consensus.cc:2932] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 2 LEADER]: Committing config change with OpId 2.195: config changed from index 151 to 195, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:21.358199 28845 raft_consensus.cc:2932] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Committing config change with OpId 2.195: config changed from index 151 to 195, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:21.361019 27444 catalog_manager.cc:5484] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 reported cstate change: config changed from index 151 to 195, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:21.362046 28978 raft_consensus.cc:1261] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32 [term 2 LEARNER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 2 index: 194. Preceding OpId from leader: term: 2 index: 195. (index mismatch)
I20240501 14:09:21.363754 29382 consensus_queue.cc:1035] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 194, Time since last communication: 0.000s
I20240501 14:09:21.366241 28979 raft_consensus.cc:2932] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32 [term 2 FOLLOWER]: Committing config change with OpId 2.195: config changed from index 151 to 195, ae4fd219bcae42e3969310879588dc32 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:21.575942 27718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fd80131af5484a938b410e65b6ea73c7"
dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59292
I20240501 14:09:21.576134 27718 raft_consensus.cc:595] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 2 LEADER]: Received request to transfer leadership
W20240501 14:09:21.589162 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.591212 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.594455 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.601583 27680 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.610078 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.616938 27680 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.628001 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.636484 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.645798 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.659142 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.674150 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.690408 27680 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.706434 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.722512 27680 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.737917 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.756366 27680 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.781131 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.800277 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.820196 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
I20240501 14:09:21.830546 27718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fd80131af5484a938b410e65b6ea73c7"
dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59298
I20240501 14:09:21.830703 27718 raft_consensus.cc:595] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 2 LEADER]: Received request to transfer leadership
W20240501 14:09:21.845441 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.870421 27682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.893675 27680 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.918924 27680 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
W20240501 14:09:21.943768 27680 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Service unavailable: leader transfer in progress
I20240501 14:09:21.956416 29350 raft_consensus.cc:979] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170: : Instructing follower 4c37ce2bb5d844c09969e644d463cf17 to start an election
I20240501 14:09:21.956538 29382 raft_consensus.cc:1067] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 2 LEADER]: Signalling peer 4c37ce2bb5d844c09969e644d463cf17 to start an election
I20240501 14:09:21.957582 28846 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fd80131af5484a938b410e65b6ea73c7"
dest_uuid: "4c37ce2bb5d844c09969e644d463cf17"
from {username='slave'} at 127.23.191.66:48007
I20240501 14:09:21.957777 28846 raft_consensus.cc:482] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:21.957962 28846 raft_consensus.cc:3037] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:09:21.959004 28846 raft_consensus.cc:504] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } }
I20240501 14:09:21.967993 27718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd80131af5484a938b410e65b6ea73c7" candidate_uuid: "4c37ce2bb5d844c09969e644d463cf17" candidate_term: 3 candidate_status { last_received { term: 2 index: 206 } } ignore_live_leader: true dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
I20240501 14:09:21.968147 27718 raft_consensus.cc:3032] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:09:21.968199 27718 raft_consensus.cc:729] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Running, Role: LEADER
I20240501 14:09:21.968346 27718 consensus_queue.cc:260] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 206, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } }
I20240501 14:09:21.968503 27718 raft_consensus.cc:3037] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:09:21.969575 27718 raft_consensus.cc:2454] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c37ce2bb5d844c09969e644d463cf17 in term 3.
I20240501 14:09:21.969944 28779 leader_election.cc:302] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c37ce2bb5d844c09969e644d463cf17, fec89cd1e3ba491eab76811aae4af170; no voters:
I20240501 14:09:21.970357 30054 raft_consensus.cc:2781] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:09:21.970518 30054 raft_consensus.cc:686] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 3 LEADER]: Becoming Leader. State: Replica: 4c37ce2bb5d844c09969e644d463cf17, State: Running, Role: LEADER
I20240501 14:09:21.971001 30054 consensus_queue.cc:237] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 206, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } }
I20240501 14:09:21.974040 27443 catalog_manager.cc:5484] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 reported cstate change: term changed from 2 to 3, leader changed from fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) to 4c37ce2bb5d844c09969e644d463cf17 (127.23.191.68). New cstate: current_term: 3 leader_uuid: "4c37ce2bb5d844c09969e644d463cf17" committed_config { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:21.976817 27680 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59086: Illegal state: replica fec89cd1e3ba491eab76811aae4af170 is not leader of this config: current role FOLLOWER
I20240501 14:09:21.981856 28846 leader_election.cc:288] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [CANDIDATE]: Term 3 election: Requested vote from peers fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265), ae4fd219bcae42e3969310879588dc32 (127.23.191.69:39239)
I20240501 14:09:21.988077 28978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd80131af5484a938b410e65b6ea73c7" candidate_uuid: "4c37ce2bb5d844c09969e644d463cf17" candidate_term: 3 candidate_status { last_received { term: 2 index: 206 } } ignore_live_leader: true dest_uuid: "ae4fd219bcae42e3969310879588dc32"
I20240501 14:09:21.988271 28978 raft_consensus.cc:3037] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:09:21.989496 28978 raft_consensus.cc:2454] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c37ce2bb5d844c09969e644d463cf17 in term 3.
I20240501 14:09:22.052007 27718 raft_consensus.cc:1261] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 3 FOLLOWER]: Refusing update from remote peer 4c37ce2bb5d844c09969e644d463cf17: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 3 index: 208. (index mismatch)
I20240501 14:09:22.052852 30055 consensus_queue.cc:1035] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20240501 14:09:22.055505 28978 raft_consensus.cc:1261] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32 [term 3 FOLLOWER]: Refusing update from remote peer 4c37ce2bb5d844c09969e644d463cf17: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 3 index: 208. (index mismatch)
I20240501 14:09:22.056028 30055 consensus_queue.cc:1035] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20240501 14:09:22.102707 28845 consensus_queue.cc:237] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 3.211, Last appended by leader: 206, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } }
I20240501 14:09:22.103981 28978 raft_consensus.cc:1261] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32 [term 3 FOLLOWER]: Refusing update from remote peer 4c37ce2bb5d844c09969e644d463cf17: Log matching property violated. Preceding OpId in replica: term: 3 index: 211. Preceding OpId from leader: term: 3 index: 212. (index mismatch)
I20240501 14:09:22.105593 30072 consensus_queue.cc:1035] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I20240501 14:09:22.109196 30054 raft_consensus.cc:2932] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 [term 3 LEADER]: Committing config change with OpId 3.212: config changed from index 195 to 212, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:22.110852 27435 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fd80131af5484a938b410e65b6ea73c7 with cas_config_opid_index 195: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:22.111709 28979 raft_consensus.cc:2932] T fd80131af5484a938b410e65b6ea73c7 P ae4fd219bcae42e3969310879588dc32 [term 3 FOLLOWER]: Committing config change with OpId 3.212: config changed from index 195 to 212, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } } }
I20240501 14:09:22.112771 27443 catalog_manager.cc:5484] T fd80131af5484a938b410e65b6ea73c7 P 4c37ce2bb5d844c09969e644d463cf17 reported cstate change: config changed from index 195 to 212, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New cstate: current_term: 3 leader_uuid: "4c37ce2bb5d844c09969e644d463cf17" committed_config { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "4c37ce2bb5d844c09969e644d463cf17" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 43375 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4fd219bcae42e3969310879588dc32" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 39239 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:22.116395 27702 tablet_service.cc:1514] Processing DeleteTablet for tablet fd80131af5484a938b410e65b6ea73c7 with delete_type TABLET_DATA_TOMBSTONED (TS fec89cd1e3ba491eab76811aae4af170 not found in new config with opid_index 212) from {username='slave'} at 127.0.0.1:48430
I20240501 14:09:22.129269 30074 tablet_replica.cc:330] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170: stopping tablet replica
I20240501 14:09:22.129455 30074 raft_consensus.cc:2229] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:22.129567 30074 pending_rounds.cc:62] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170: Trying to abort 1 pending ops.
I20240501 14:09:22.129606 30074 pending_rounds.cc:68] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170: Aborting op as it isn't in flight: id { term: 3 index: 211 } timestamp: 7022889214352658432 op_type: WRITE_OP write_request { tablet_id: "fd80131af5484a938b410e65b6ea73c7" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000W\262\214\013\275\204\032\306\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""6820369135bcfc57" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022889214320926720 authz_token { token_data: "\010\270\234\311\261\006\"3\n\005slave\022*\n 7230a83999e24b5fb8afb43b5c8944e4\020\001\030\001 \001(\001" signature: "<redacted>""K5^rG\251\357(t\311\032\260@\001b\356.X\240\347\230(\201\035\271@\004\265\302m\233\313\014s\251\334\027\036\256x\262y\016i!\314\224J(\304t\203\261L\226\350\324\n\377\004\332\020\351\360" signing_key_seq_num: 0 } } request_id { client_id: "7fa5ca6b433c44d78f7795e3504531e7" seq_no: 585 first_incomplete_seq_no: 585 attempt_no: 0 }
I20240501 14:09:22.129885 30074 raft_consensus.cc:2258] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:22.132033 30074 ts_tablet_manager.cc:1875] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:22.134619 30074 ts_tablet_manager.cc:1888] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.211
I20240501 14:09:22.134688 30074 log.cc:1196] T fd80131af5484a938b410e65b6ea73c7 P fec89cd1e3ba491eab76811aae4af170: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/fd80131af5484a938b410e65b6ea73c7
I20240501 14:09:22.135360 27434 catalog_manager.cc:4830] TS fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265): tablet fd80131af5484a938b410e65b6ea73c7 (table rebalance_test_table_4 [id=7230a83999e24b5fb8afb43b5c8944e4]) successfully deleted
I20240501 14:09:22.501792 29091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:22.506003 28959 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:22.513901 27832 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:22.535916 27702 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:22.536221 29224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:22.559463 27570 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:22.581799 28826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:22.891924 29091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:22.896353 27832 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:22.906585 28959 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:22.917801 27570 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:22.918911 29224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:22.925891 28826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:22.945818 27702 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:23.582870 27570 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:23.584648 28826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:23.589273 28959 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:23.591640 27832 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:23.592998 29091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:23.600608 29224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:23.626610 27702 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:23.930055 28959 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:23.935261 28826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:23.940855 27832 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:23.941820 27702 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:23.962776 27570 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:23.987349 29224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:24.018882 29091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:24.203222 28959 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:24.203610 29091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:24.204006 28826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:24.204761 27570 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:24.208998 27702 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:24.225478 27832 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:24.235347 29224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
25f42fba200742378c49fb096885390d | 127.23.191.126:38485 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.191.84:38191 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
3625e18e2ea14d43aa3b86b014fd89f5 | 127.23.191.70:33907 | HEALTHY | <none> | 1 | 0
4c37ce2bb5d844c09969e644d463cf17 | 127.23.191.68:43375 | HEALTHY | <none> | 1 | 0
ae4fd219bcae42e3969310879588dc32 | 127.23.191.69:39239 | HEALTHY | <none> | 0 | 0
c9cc3f1efaf44a4895d3109beabe5312 | 127.23.191.65:34669 | HEALTHY | <none> | 4 | 0
d1f8294a92e74484ab4a653a4ecf84e5 | 127.23.191.67:45647 | HEALTHY | <none> | 5 | 0
fec89cd1e3ba491eab76811aae4af170 | 127.23.191.66:44265 | HEALTHY | <none> | 5 | 0
ff8b0a555ccd48bcb11dca38d3b2f2f7 | 127.23.191.71:33963 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.23.191.65 | experimental | 127.23.191.65:34669
local_ip_for_outbound_sockets | 127.23.191.66 | experimental | 127.23.191.66:44265
local_ip_for_outbound_sockets | 127.23.191.67 | experimental | 127.23.191.67:45647
local_ip_for_outbound_sockets | 127.23.191.68 | experimental | 127.23.191.68:43375
local_ip_for_outbound_sockets | 127.23.191.69 | experimental | 127.23.191.69:39239
local_ip_for_outbound_sockets | 127.23.191.70 | experimental | 127.23.191.70:33907
local_ip_for_outbound_sockets | 127.23.191.71 | experimental | 127.23.191.71:33963
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.191.65:34669
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.23.191.66:44265
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.191.67:45647
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.23.191.68:43375
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.23.191.69:39239
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.23.191.70:33907
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.23.191.71:33963
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.191.84:38191 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:09:24.258136 24317 log_verifier.cc:126] Checking tablet 085c3852d1ee47dba35aa465b3e36337
I20240501 14:09:24.308948 24317 log_verifier.cc:177] Verified matching terms for 266 ops in tablet 085c3852d1ee47dba35aa465b3e36337
I20240501 14:09:24.309114 24317 log_verifier.cc:126] Checking tablet 322635d2e89b4bd69e1f1442b0f811e8
I20240501 14:09:24.360885 24317 log_verifier.cc:177] Verified matching terms for 273 ops in tablet 322635d2e89b4bd69e1f1442b0f811e8
I20240501 14:09:24.361008 24317 log_verifier.cc:126] Checking tablet 41d8c8dbc45b449583f0a1808c634855
I20240501 14:09:24.413972 24317 log_verifier.cc:177] Verified matching terms for 286 ops in tablet 41d8c8dbc45b449583f0a1808c634855
I20240501 14:09:24.414093 24317 log_verifier.cc:126] Checking tablet 78612cef5af34df3b33567744725af28
I20240501 14:09:24.427462 24317 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 78612cef5af34df3b33567744725af28
I20240501 14:09:24.427523 24317 log_verifier.cc:126] Checking tablet 8a5ec8adfb714bf98dcf3279b9df2148
I20240501 14:09:24.483193 24317 log_verifier.cc:177] Verified matching terms for 251 ops in tablet 8a5ec8adfb714bf98dcf3279b9df2148
I20240501 14:09:24.483316 24317 log_verifier.cc:126] Checking tablet 8cf4c3bcfc524e708092cd92d0622684
I20240501 14:09:24.537330 24317 log_verifier.cc:177] Verified matching terms for 295 ops in tablet 8cf4c3bcfc524e708092cd92d0622684
I20240501 14:09:24.537459 24317 log_verifier.cc:126] Checking tablet ad72089b2cf54cd0b61ee8bb2c95975f
I20240501 14:09:24.589093 24317 log_verifier.cc:177] Verified matching terms for 246 ops in tablet ad72089b2cf54cd0b61ee8bb2c95975f
I20240501 14:09:24.589237 24317 log_verifier.cc:126] Checking tablet b54ea6caa4f6448790b5456a2e5676c2
I20240501 14:09:24.648273 24317 log_verifier.cc:177] Verified matching terms for 284 ops in tablet b54ea6caa4f6448790b5456a2e5676c2
I20240501 14:09:24.648415 24317 log_verifier.cc:126] Checking tablet c717dce4871e459390c508e275c52b31
I20240501 14:09:24.699136 24317 log_verifier.cc:177] Verified matching terms for 275 ops in tablet c717dce4871e459390c508e275c52b31
I20240501 14:09:24.699254 24317 log_verifier.cc:126] Checking tablet ca6f19b8716843ccaabf666651584928
I20240501 14:09:24.751075 24317 log_verifier.cc:177] Verified matching terms for 268 ops in tablet ca6f19b8716843ccaabf666651584928
I20240501 14:09:24.751191 24317 log_verifier.cc:126] Checking tablet dcbe4acb729e48bb89d5a26cb211b2eb
I20240501 14:09:24.806953 24317 log_verifier.cc:177] Verified matching terms for 306 ops in tablet dcbe4acb729e48bb89d5a26cb211b2eb
I20240501 14:09:24.807077 24317 log_verifier.cc:126] Checking tablet e06d6b042b604fd19f19c44c3161beb8
I20240501 14:09:24.859360 24317 log_verifier.cc:177] Verified matching terms for 284 ops in tablet e06d6b042b604fd19f19c44c3161beb8
I20240501 14:09:24.859479 24317 log_verifier.cc:126] Checking tablet e0d87c63b9f64212a3b451bb4140979e
I20240501 14:09:24.915953 24317 log_verifier.cc:177] Verified matching terms for 321 ops in tablet e0d87c63b9f64212a3b451bb4140979e
I20240501 14:09:24.916225 24317 log_verifier.cc:126] Checking tablet ea2e1d9ab9a54d52b89c72c4ef442ce5
I20240501 14:09:24.966902 24317 log_verifier.cc:177] Verified matching terms for 259 ops in tablet ea2e1d9ab9a54d52b89c72c4ef442ce5
I20240501 14:09:24.967015 24317 log_verifier.cc:126] Checking tablet fd80131af5484a938b410e65b6ea73c7
I20240501 14:09:25.028870 24317 log_verifier.cc:177] Verified matching terms for 300 ops in tablet fd80131af5484a938b410e65b6ea73c7
I20240501 14:09:25.028996 24317 log_verifier.cc:126] Checking tablet fe7fe0171f2d477c91661a766239b9fa
I20240501 14:09:25.077821 24317 log_verifier.cc:177] Verified matching terms for 244 ops in tablet fe7fe0171f2d477c91661a766239b9fa
I20240501 14:09:25.079808 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.72:0
--local_ip_for_outbound_sockets=127.23.191.72
--webserver_interface=127.23.191.72
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--builtin_ntp_servers=127.23.191.84:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:25.195959 30289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:25.196180 30289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:25.196223 30289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:25.199188 30289 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:25.199254 30289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:25.199321 30289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.72
I20240501 14:09:25.202905 30289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.23.191.72
--webserver_port=0
--tserver_master_addrs=127.23.191.126:38485
--never_fsync=true
--heap_profile_path=/tmp/kudu.30289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.72
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:25.203790 30289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:25.204932 30289 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:25.213182 30295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:25.214031 30296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:25.214105 30298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:25.214579 30289 server_base.cc:1034] running on GCE node
I20240501 14:09:25.214926 30289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:25.215456 30289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:25.216595 30289 hybrid_clock.cc:648] HybridClock initialized: now 1714572565216580 us; error 35 us; skew 500 ppm
I20240501 14:09:25.218333 30289 webserver.cc:460] Webserver started at http://127.23.191.72:44593/ using document root <none> and password file <none>
I20240501 14:09:25.218853 30289 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:25.218907 30289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:25.219089 30289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:25.220752 30289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "67eb982d9330483183c06df51449cdbd"
format_stamp: "Formatted at 2024-05-01 14:09:25 on dist-test-slave-c4g3"
I20240501 14:09:25.221292 30289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "67eb982d9330483183c06df51449cdbd"
format_stamp: "Formatted at 2024-05-01 14:09:25 on dist-test-slave-c4g3"
I20240501 14:09:25.224476 30289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:25.226632 30304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:25.227614 30289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:25.227715 30289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "67eb982d9330483183c06df51449cdbd"
format_stamp: "Formatted at 2024-05-01 14:09:25 on dist-test-slave-c4g3"
I20240501 14:09:25.227793 30289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:25.249173 30289 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:25.249864 30289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:25.250025 30289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:25.250550 30289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:25.251492 30289 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:25.251539 30289 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:25.251590 30289 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:25.251621 30289 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:25.260299 30289 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.72:36075
I20240501 14:09:25.260318 30417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.72:36075 every 8 connection(s)
I20240501 14:09:25.261256 30289 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:09:25.266626 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 30289
I20240501 14:09:25.266739 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:09:25.269893 30418 heartbeater.cc:347] Connected to a master server at 127.23.191.126:38485
I20240501 14:09:25.270129 30418 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:25.270665 30418 heartbeater.cc:511] Master 127.23.191.126:38485 requested a full tablet report, sending...
I20240501 14:09:25.271679 27447 ts_manager.cc:194] Registered new tserver with Master: 67eb982d9330483183c06df51449cdbd (127.23.191.72:36075)
I20240501 14:09:25.272235 27447 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.72:37103
I20240501 14:09:25.456029 28959 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:25.457636 29091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:25.458106 28826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:25.461899 27702 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:25.464411 30352 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:25.469713 29224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:25.470870 27570 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:25.471330 27832 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:25.570044 30352 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:25.573843 28826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:25.574970 29091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:25.583380 28959 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:25.589953 27570 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:25.590224 27702 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:25.602236 27832 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:25.609294 29224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:25.644331 27853 consensus_queue.cc:237] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 1.251, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } }
I20240501 14:09:25.645740 29241 raft_consensus.cc:1261] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 251. Preceding OpId from leader: term: 1 index: 252. (index mismatch)
I20240501 14:09:25.646126 29360 consensus_queue.cc:1035] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s
I20240501 14:09:25.647866 30468 raft_consensus.cc:2932] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.252: config changed from index 76 to 252, NON_VOTER 67eb982d9330483183c06df51449cdbd (127.23.191.72) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } } }
I20240501 14:09:25.648345 29241 raft_consensus.cc:2932] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Committing config change with OpId 1.252: config changed from index 76 to 252, NON_VOTER 67eb982d9330483183c06df51449cdbd (127.23.191.72) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } } }
I20240501 14:09:25.650537 27449 catalog_manager.cc:5484] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index 76 to 252, NON_VOTER 67eb982d9330483183c06df51449cdbd (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:25.651430 27787 consensus_peers.cc:485] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 -> Peer 67eb982d9330483183c06df51449cdbd (127.23.191.72:36075): Couldn't send request to peer 67eb982d9330483183c06df51449cdbd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a5ec8adfb714bf98dcf3279b9df2148. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:25.657757 27853 consensus_queue.cc:237] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 1.244, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } }
W20240501 14:09:25.659075 27787 consensus_peers.cc:485] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 -> Peer 67eb982d9330483183c06df51449cdbd (127.23.191.72:36075): Couldn't send request to peer 67eb982d9330483183c06df51449cdbd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe7fe0171f2d477c91661a766239b9fa. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:25.659116 29110 raft_consensus.cc:1261] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 245. (index mismatch)
I20240501 14:09:25.659440 29360 consensus_queue.cc:1035] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20240501 14:09:25.661115 30468 raft_consensus.cc:2932] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.245: config changed from index 80 to 245, NON_VOTER 67eb982d9330483183c06df51449cdbd (127.23.191.72) added. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } } }
I20240501 14:09:25.661525 29110 raft_consensus.cc:2932] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Committing config change with OpId 1.245: config changed from index 80 to 245, NON_VOTER 67eb982d9330483183c06df51449cdbd (127.23.191.72) added. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } } }
I20240501 14:09:25.664263 27447 catalog_manager.cc:5484] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index 80 to 245, NON_VOTER 67eb982d9330483183c06df51449cdbd (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:25.678478 27590 consensus_queue.cc:237] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 1.286, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } }
I20240501 14:09:25.680019 29241 raft_consensus.cc:1261] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 286. Preceding OpId from leader: term: 1 index: 287. (index mismatch)
I20240501 14:09:25.680469 30535 consensus_queue.cc:1035] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20240501 14:09:25.682258 30290 raft_consensus.cc:2932] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.287: config changed from index 80 to 287, NON_VOTER 67eb982d9330483183c06df51449cdbd (127.23.191.72) added. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } } }
I20240501 14:09:25.684962 27449 catalog_manager.cc:5484] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index 80 to 287, NON_VOTER 67eb982d9330483183c06df51449cdbd (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:25.685180 27525 consensus_peers.cc:485] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 -> Peer 67eb982d9330483183c06df51449cdbd (127.23.191.72:36075): Couldn't send request to peer 67eb982d9330483183c06df51449cdbd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41d8c8dbc45b449583f0a1808c634855. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:25.685423 29241 raft_consensus.cc:2932] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Committing config change with OpId 1.287: config changed from index 80 to 287, NON_VOTER 67eb982d9330483183c06df51449cdbd (127.23.191.72) added. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } } }
I20240501 14:09:25.693562 27718 consensus_queue.cc:237] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 1.266, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } }
I20240501 14:09:25.694944 27590 raft_consensus.cc:1261] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20240501 14:09:25.695399 29382 consensus_queue.cc:1035] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
W20240501 14:09:25.698035 27656 consensus_peers.cc:485] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 -> Peer 67eb982d9330483183c06df51449cdbd (127.23.191.72:36075): Couldn't send request to peer 67eb982d9330483183c06df51449cdbd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 085c3852d1ee47dba35aa465b3e36337. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:25.701356 29382 raft_consensus.cc:2932] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Committing config change with OpId 1.267: config changed from index -1 to 267, NON_VOTER 67eb982d9330483183c06df51449cdbd (127.23.191.72) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } } }
I20240501 14:09:25.704305 27449 catalog_manager.cc:5484] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 reported cstate change: config changed from index -1 to 267, NON_VOTER 67eb982d9330483183c06df51449cdbd (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:25.704957 27590 raft_consensus.cc:2932] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index -1 to 267, NON_VOTER 67eb982d9330483183c06df51449cdbd (127.23.191.72) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } } }
I20240501 14:09:26.034060 30588 ts_tablet_manager.cc:902] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd: Initiating tablet copy from peer d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:26.034735 30588 tablet_copy_client.cc:308] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:45647
I20240501 14:09:26.038568 27872 tablet_copy_service.cc:140] P d1f8294a92e74484ab4a653a4ecf84e5: Received BeginTabletCopySession request for tablet 8a5ec8adfb714bf98dcf3279b9df2148 from peer 67eb982d9330483183c06df51449cdbd ({username='slave'} at 127.23.191.72:57155)
I20240501 14:09:26.038682 27872 tablet_copy_service.cc:161] P d1f8294a92e74484ab4a653a4ecf84e5: Beginning new tablet copy session on tablet 8a5ec8adfb714bf98dcf3279b9df2148 from peer 67eb982d9330483183c06df51449cdbd at {username='slave'} at 127.23.191.72:57155: session id = 67eb982d9330483183c06df51449cdbd-8a5ec8adfb714bf98dcf3279b9df2148
I20240501 14:09:26.039408 27872 tablet_copy_source_session.cc:205] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:26.042340 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5ec8adfb714bf98dcf3279b9df2148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.047046 30588 tablet_copy_client.cc:783] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:26.047235 30588 tablet_copy_client.cc:647] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:26.049484 30588 tablet_copy_client.cc:523] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:26.051072 30588 tablet_bootstrap.cc:492] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd: Bootstrap starting.
I20240501 14:09:26.069370 30588 log.cc:824] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:26.111850 30595 ts_tablet_manager.cc:902] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd: Initiating tablet copy from peer c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669)
I20240501 14:09:26.112360 30595 tablet_copy_client.cc:308] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:34669
I20240501 14:09:26.115870 27611 tablet_copy_service.cc:140] P c9cc3f1efaf44a4895d3109beabe5312: Received BeginTabletCopySession request for tablet 41d8c8dbc45b449583f0a1808c634855 from peer 67eb982d9330483183c06df51449cdbd ({username='slave'} at 127.23.191.72:57913)
I20240501 14:09:26.115968 27611 tablet_copy_service.cc:161] P c9cc3f1efaf44a4895d3109beabe5312: Beginning new tablet copy session on tablet 41d8c8dbc45b449583f0a1808c634855 from peer 67eb982d9330483183c06df51449cdbd at {username='slave'} at 127.23.191.72:57913: session id = 67eb982d9330483183c06df51449cdbd-41d8c8dbc45b449583f0a1808c634855
I20240501 14:09:26.116813 27611 tablet_copy_source_session.cc:205] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:26.117660 30595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d8c8dbc45b449583f0a1808c634855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.119865 30595 tablet_copy_client.cc:783] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:26.120043 30595 tablet_copy_client.cc:647] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:26.122095 30595 tablet_copy_client.cc:523] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:26.123261 30595 tablet_bootstrap.cc:492] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd: Bootstrap starting.
I20240501 14:09:26.174569 30621 ts_tablet_manager.cc:902] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd: Initiating tablet copy from peer d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647)
I20240501 14:09:26.175062 30621 tablet_copy_client.cc:308] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:45647
I20240501 14:09:26.175491 27872 tablet_copy_service.cc:140] P d1f8294a92e74484ab4a653a4ecf84e5: Received BeginTabletCopySession request for tablet fe7fe0171f2d477c91661a766239b9fa from peer 67eb982d9330483183c06df51449cdbd ({username='slave'} at 127.23.191.72:57155)
I20240501 14:09:26.175586 27872 tablet_copy_service.cc:161] P d1f8294a92e74484ab4a653a4ecf84e5: Beginning new tablet copy session on tablet fe7fe0171f2d477c91661a766239b9fa from peer 67eb982d9330483183c06df51449cdbd at {username='slave'} at 127.23.191.72:57155: session id = 67eb982d9330483183c06df51449cdbd-fe7fe0171f2d477c91661a766239b9fa
I20240501 14:09:26.176299 27872 tablet_copy_source_session.cc:205] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:26.177054 30621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7fe0171f2d477c91661a766239b9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.179800 30621 tablet_copy_client.cc:783] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:26.179945 30621 tablet_copy_client.cc:647] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:26.181447 30621 tablet_copy_client.cc:523] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:26.182425 30621 tablet_bootstrap.cc:492] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd: Bootstrap starting.
I20240501 14:09:26.229001 30588 tablet_bootstrap.cc:492] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=252 overwritten=0 applied=252 ignored=0} inserts{seen=921 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:26.229440 30588 tablet_bootstrap.cc:492] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd: Bootstrap complete.
I20240501 14:09:26.229835 30588 ts_tablet_manager.cc:1367] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd: Time spent bootstrapping tablet: real 0.179s user 0.170s sys 0.008s
I20240501 14:09:26.232549 30588 raft_consensus.cc:348] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } }
I20240501 14:09:26.232846 30588 raft_consensus.cc:729] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67eb982d9330483183c06df51449cdbd, State: Initialized, Role: LEARNER
I20240501 14:09:26.233213 30588 consensus_queue.cc:260] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 252, Last appended: 1.252, Last appended by leader: 252, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } }
I20240501 14:09:26.233992 30418 heartbeater.cc:503] Master 127.23.191.126:38485 was elected leader, sending a full tablet report...
I20240501 14:09:26.235359 30588 ts_tablet_manager.cc:1398] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:09:26.235932 27872 tablet_copy_service.cc:342] P d1f8294a92e74484ab4a653a4ecf84e5: Request end of tablet copy session 67eb982d9330483183c06df51449cdbd-8a5ec8adfb714bf98dcf3279b9df2148 received from {username='slave'} at 127.23.191.72:57155
I20240501 14:09:26.236011 27872 tablet_copy_service.cc:433] P d1f8294a92e74484ab4a653a4ecf84e5: ending tablet copy session 67eb982d9330483183c06df51449cdbd-8a5ec8adfb714bf98dcf3279b9df2148 on tablet 8a5ec8adfb714bf98dcf3279b9df2148 with peer 67eb982d9330483183c06df51449cdbd
I20240501 14:09:26.239576 30588 ts_tablet_manager.cc:902] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd: Initiating tablet copy from peer fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:26.240013 30588 tablet_copy_client.cc:308] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:44265
I20240501 14:09:26.244464 27742 tablet_copy_service.cc:140] P fec89cd1e3ba491eab76811aae4af170: Received BeginTabletCopySession request for tablet 085c3852d1ee47dba35aa465b3e36337 from peer 67eb982d9330483183c06df51449cdbd ({username='slave'} at 127.23.191.72:48839)
I20240501 14:09:26.244587 27742 tablet_copy_service.cc:161] P fec89cd1e3ba491eab76811aae4af170: Beginning new tablet copy session on tablet 085c3852d1ee47dba35aa465b3e36337 from peer 67eb982d9330483183c06df51449cdbd at {username='slave'} at 127.23.191.72:48839: session id = 67eb982d9330483183c06df51449cdbd-085c3852d1ee47dba35aa465b3e36337
I20240501 14:09:26.245571 27742 tablet_copy_source_session.cc:205] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:26.246398 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 085c3852d1ee47dba35aa465b3e36337. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.248518 30588 tablet_copy_client.cc:783] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:26.248663 30588 tablet_copy_client.cc:647] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:26.250938 30588 tablet_copy_client.cc:523] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:26.252060 30588 tablet_bootstrap.cc:492] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd: Bootstrap starting.
I20240501 14:09:26.340329 30595 tablet_bootstrap.cc:492] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=287 overwritten=0 applied=287 ignored=0} inserts{seen=915 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:26.340677 30595 tablet_bootstrap.cc:492] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd: Bootstrap complete.
I20240501 14:09:26.340920 30595 ts_tablet_manager.cc:1367] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd: Time spent bootstrapping tablet: real 0.218s user 0.194s sys 0.021s
I20240501 14:09:26.341586 30595 raft_consensus.cc:348] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } }
I20240501 14:09:26.341722 30595 raft_consensus.cc:729] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67eb982d9330483183c06df51449cdbd, State: Initialized, Role: LEARNER
I20240501 14:09:26.341832 30595 consensus_queue.cc:260] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 287, Last appended: 1.287, Last appended by leader: 287, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } }
I20240501 14:09:26.342134 30595 ts_tablet_manager.cc:1398] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:26.342921 27611 tablet_copy_service.cc:342] P c9cc3f1efaf44a4895d3109beabe5312: Request end of tablet copy session 67eb982d9330483183c06df51449cdbd-41d8c8dbc45b449583f0a1808c634855 received from {username='slave'} at 127.23.191.72:57913
I20240501 14:09:26.343039 27611 tablet_copy_service.cc:433] P c9cc3f1efaf44a4895d3109beabe5312: ending tablet copy session 67eb982d9330483183c06df51449cdbd-41d8c8dbc45b449583f0a1808c634855 on tablet 41d8c8dbc45b449583f0a1808c634855 with peer 67eb982d9330483183c06df51449cdbd
I20240501 14:09:26.364532 30621 tablet_bootstrap.cc:492] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=245 overwritten=0 applied=245 ignored=0} inserts{seen=914 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:26.364789 30621 tablet_bootstrap.cc:492] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd: Bootstrap complete.
I20240501 14:09:26.364985 30621 ts_tablet_manager.cc:1367] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd: Time spent bootstrapping tablet: real 0.183s user 0.160s sys 0.008s
I20240501 14:09:26.365384 30621 raft_consensus.cc:348] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } }
I20240501 14:09:26.365486 30621 raft_consensus.cc:729] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67eb982d9330483183c06df51449cdbd, State: Initialized, Role: LEARNER
I20240501 14:09:26.365561 30621 consensus_queue.cc:260] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 245, Last appended: 1.245, Last appended by leader: 245, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } }
I20240501 14:09:26.365810 30621 ts_tablet_manager.cc:1398] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:26.366372 27872 tablet_copy_service.cc:342] P d1f8294a92e74484ab4a653a4ecf84e5: Request end of tablet copy session 67eb982d9330483183c06df51449cdbd-fe7fe0171f2d477c91661a766239b9fa received from {username='slave'} at 127.23.191.72:57155
I20240501 14:09:26.366468 27872 tablet_copy_service.cc:433] P d1f8294a92e74484ab4a653a4ecf84e5: ending tablet copy session 67eb982d9330483183c06df51449cdbd-fe7fe0171f2d477c91661a766239b9fa on tablet fe7fe0171f2d477c91661a766239b9fa with peer 67eb982d9330483183c06df51449cdbd
I20240501 14:09:26.419279 30588 tablet_bootstrap.cc:492] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=267 overwritten=0 applied=267 ignored=0} inserts{seen=952 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:26.419523 30588 tablet_bootstrap.cc:492] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd: Bootstrap complete.
I20240501 14:09:26.419672 30588 ts_tablet_manager.cc:1367] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd: Time spent bootstrapping tablet: real 0.168s user 0.156s sys 0.008s
I20240501 14:09:26.420038 30588 raft_consensus.cc:348] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } }
I20240501 14:09:26.420141 30588 raft_consensus.cc:729] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67eb982d9330483183c06df51449cdbd, State: Initialized, Role: LEARNER
I20240501 14:09:26.420219 30588 consensus_queue.cc:260] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 267, Last appended: 1.267, Last appended by leader: 267, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: true } }
I20240501 14:09:26.420429 30588 ts_tablet_manager.cc:1398] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:26.421183 27742 tablet_copy_service.cc:342] P fec89cd1e3ba491eab76811aae4af170: Request end of tablet copy session 67eb982d9330483183c06df51449cdbd-085c3852d1ee47dba35aa465b3e36337 received from {username='slave'} at 127.23.191.72:48839
I20240501 14:09:26.421265 27742 tablet_copy_service.cc:433] P fec89cd1e3ba491eab76811aae4af170: ending tablet copy session 67eb982d9330483183c06df51449cdbd-085c3852d1ee47dba35aa465b3e36337 on tablet 085c3852d1ee47dba35aa465b3e36337 with peer 67eb982d9330483183c06df51449cdbd
I20240501 14:09:26.537786 30372 raft_consensus.cc:1203] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.286->[1.287-1.287] Dedup: 1.287->[]
I20240501 14:09:26.569430 30372 raft_consensus.cc:1203] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.251->[1.252-1.252] Dedup: 1.252->[]
I20240501 14:09:26.735556 30372 raft_consensus.cc:1203] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.244->[1.245-1.245] Dedup: 1.245->[]
I20240501 14:09:26.754940 30372 raft_consensus.cc:1203] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.266->[1.267-1.267] Dedup: 1.267->[]
I20240501 14:09:26.961714 30656 raft_consensus.cc:1050] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312: attempting to promote NON_VOTER 67eb982d9330483183c06df51449cdbd to VOTER
I20240501 14:09:26.962097 30656 consensus_queue.cc:237] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 1.287, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:26.963416 29241 raft_consensus.cc:1261] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 287. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20240501 14:09:26.963502 30372 raft_consensus.cc:1261] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Refusing update from remote peer c9cc3f1efaf44a4895d3109beabe5312: Log matching property violated. Preceding OpId in replica: term: 1 index: 287. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20240501 14:09:26.963860 30656 consensus_queue.cc:1035] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20240501 14:09:26.964046 30535 consensus_queue.cc:1035] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20240501 14:09:26.965761 30656 raft_consensus.cc:2932] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Committing config change with OpId 1.288: config changed from index 287 to 288, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:26.966162 29241 raft_consensus.cc:2932] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 287 to 288, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:26.967170 30372 raft_consensus.cc:2932] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 287 to 288, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:26.969797 27447 catalog_manager.cc:5484] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index 287 to 288, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9cc3f1efaf44a4895d3109beabe5312" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:26.970813 30657 raft_consensus.cc:1050] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5: attempting to promote NON_VOTER 67eb982d9330483183c06df51449cdbd to VOTER
I20240501 14:09:26.971232 30657 consensus_queue.cc:237] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 1.252, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:26.972383 30372 raft_consensus.cc:1261] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240501 14:09:26.972415 29241 raft_consensus.cc:1261] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240501 14:09:26.972903 30658 consensus_queue.cc:1035] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240501 14:09:26.973215 30658 consensus_queue.cc:1035] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240501 14:09:26.974663 30658 raft_consensus.cc:2932] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.253: config changed from index 252 to 253, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:26.975744 30372 raft_consensus.cc:2932] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 252 to 253, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:26.976547 29241 raft_consensus.cc:2932] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 252 to 253, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:26.977007 27444 catalog_manager.cc:5484] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index 252 to 253, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:27.053098 27590 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "41d8c8dbc45b449583f0a1808c634855"
dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44620
I20240501 14:09:27.053284 27590 raft_consensus.cc:595] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:27.062444 27853 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a5ec8adfb714bf98dcf3279b9df2148"
dest_uuid: "d1f8294a92e74484ab4a653a4ecf84e5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45952
I20240501 14:09:27.062572 27853 raft_consensus.cc:595] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:27.125541 30658 raft_consensus.cc:1050] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5: attempting to promote NON_VOTER 67eb982d9330483183c06df51449cdbd to VOTER
I20240501 14:09:27.125941 30658 consensus_queue.cc:237] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 1.245, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:27.127219 30372 raft_consensus.cc:1261] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 246. (index mismatch)
I20240501 14:09:27.127239 29110 raft_consensus.cc:1261] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 246. (index mismatch)
I20240501 14:09:27.127609 30658 consensus_queue.cc:1035] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20240501 14:09:27.127794 30657 consensus_queue.cc:1035] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20240501 14:09:27.129550 30658 raft_consensus.cc:2932] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.246: config changed from index 245 to 246, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:27.130286 30372 raft_consensus.cc:2932] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd [term 1 FOLLOWER]: Committing config change with OpId 1.246: config changed from index 245 to 246, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:27.132452 27447 catalog_manager.cc:5484] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index 245 to 246, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:27.133044 29110 raft_consensus.cc:2932] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Committing config change with OpId 1.246: config changed from index 245 to 246, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "3625e18e2ea14d43aa3b86b014fd89f5" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 33907 } attrs { promote: false replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:27.135324 27853 consensus_queue.cc:237] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 1.246, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:27.136296 30372 raft_consensus.cc:1261] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd [term 1 FOLLOWER]: Refusing update from remote peer d1f8294a92e74484ab4a653a4ecf84e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 246. Preceding OpId from leader: term: 1 index: 247. (index mismatch)
I20240501 14:09:27.136629 30658 consensus_queue.cc:1035] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20240501 14:09:27.138093 30468 raft_consensus.cc:2932] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Committing config change with OpId 1.247: config changed from index 246 to 247, VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:27.138525 30372 raft_consensus.cc:2932] T fe7fe0171f2d477c91661a766239b9fa P 67eb982d9330483183c06df51449cdbd [term 1 FOLLOWER]: Committing config change with OpId 1.247: config changed from index 246 to 247, VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:27.139277 27436 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe7fe0171f2d477c91661a766239b9fa with cas_config_opid_index 246: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:27.140173 27444 catalog_manager.cc:5484] T fe7fe0171f2d477c91661a766239b9fa P d1f8294a92e74484ab4a653a4ecf84e5 reported cstate change: config changed from index 246 to 247, VOTER 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70) evicted. New cstate: current_term: 1 leader_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" committed_config { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:27.144937 30680 raft_consensus.cc:1050] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170: attempting to promote NON_VOTER 67eb982d9330483183c06df51449cdbd to VOTER
I20240501 14:09:27.145345 30680 consensus_queue.cc:237] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 1.267, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:27.145921 29091 tablet_service.cc:1514] Processing DeleteTablet for tablet fe7fe0171f2d477c91661a766239b9fa with delete_type TABLET_DATA_TOMBSTONED (TS 3625e18e2ea14d43aa3b86b014fd89f5 not found in new config with opid_index 247) from {username='slave'} at 127.0.0.1:57400
I20240501 14:09:27.146575 27590 raft_consensus.cc:1261] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
I20240501 14:09:27.146572 30749 tablet_replica.cc:330] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5: stopping tablet replica
I20240501 14:09:27.146649 30372 raft_consensus.cc:1261] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [term 1 LEARNER]: Refusing update from remote peer fec89cd1e3ba491eab76811aae4af170: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
I20240501 14:09:27.147010 30749 raft_consensus.cc:2229] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:27.146981 29382 consensus_queue.cc:1035] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20240501 14:09:27.147248 30749 raft_consensus.cc:2258] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:27.148176 29382 consensus_queue.cc:1035] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20240501 14:09:27.149259 30749 ts_tablet_manager.cc:1875] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:27.149348 30680 raft_consensus.cc:2932] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:27.149907 27590 raft_consensus.cc:2932] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:27.150425 30372 raft_consensus.cc:2932] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:27.151826 30749 ts_tablet_manager.cc:1888] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.246
I20240501 14:09:27.151921 30749 log.cc:1196] T fe7fe0171f2d477c91661a766239b9fa P 3625e18e2ea14d43aa3b86b014fd89f5: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal/wals/fe7fe0171f2d477c91661a766239b9fa
I20240501 14:09:27.152277 27444 catalog_manager.cc:5484] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 reported cstate change: config changed from index 267 to 268, 67eb982d9330483183c06df51449cdbd (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fec89cd1e3ba491eab76811aae4af170" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:27.152428 27435 catalog_manager.cc:4830] TS 3625e18e2ea14d43aa3b86b014fd89f5 (127.23.191.70:33907): tablet fe7fe0171f2d477c91661a766239b9fa (table rebalance_test_table_1 [id=4f58d714ee014c97a0e588bae9a3e9c3]) successfully deleted
I20240501 14:09:27.277720 27718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "085c3852d1ee47dba35aa465b3e36337"
dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44954
I20240501 14:09:27.277860 27718 raft_consensus.cc:595] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:27.286756 27590 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "41d8c8dbc45b449583f0a1808c634855"
dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44630
I20240501 14:09:27.286882 27590 raft_consensus.cc:595] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:27.295754 27853 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a5ec8adfb714bf98dcf3279b9df2148"
dest_uuid: "d1f8294a92e74484ab4a653a4ecf84e5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45972
I20240501 14:09:27.295883 27853 raft_consensus.cc:595] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:27.309441 27718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "085c3852d1ee47dba35aa465b3e36337"
dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44966
I20240501 14:09:27.309567 27718 raft_consensus.cc:595] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:27.318122 27590 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "41d8c8dbc45b449583f0a1808c634855"
dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44654
I20240501 14:09:27.318254 27590 raft_consensus.cc:595] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:27.327523 27853 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a5ec8adfb714bf98dcf3279b9df2148"
dest_uuid: "d1f8294a92e74484ab4a653a4ecf84e5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46006
I20240501 14:09:27.327651 27853 raft_consensus.cc:595] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:27.350210 30655 raft_consensus.cc:979] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312: : Instructing follower ff8b0a555ccd48bcb11dca38d3b2f2f7 to start an election
I20240501 14:09:27.350320 30655 raft_consensus.cc:1067] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Signalling peer ff8b0a555ccd48bcb11dca38d3b2f2f7 to start an election
I20240501 14:09:27.350759 29241 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "41d8c8dbc45b449583f0a1808c634855"
dest_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7"
from {username='slave'} at 127.23.191.65:42299
I20240501 14:09:27.350932 29241 raft_consensus.cc:482] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:27.350982 29241 raft_consensus.cc:3037] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:27.351758 29241 raft_consensus.cc:504] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:27.352674 29241 leader_election.cc:288] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [CANDIDATE]: Term 2 election: Requested vote from peers c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669), 67eb982d9330483183c06df51449cdbd (127.23.191.72:36075)
I20240501 14:09:27.352953 27590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d8c8dbc45b449583f0a1808c634855" candidate_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 288 } } ignore_live_leader: true dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
I20240501 14:09:27.353070 27590 raft_consensus.cc:3032] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:27.353127 27590 raft_consensus.cc:729] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9cc3f1efaf44a4895d3109beabe5312, State: Running, Role: LEADER
I20240501 14:09:27.353260 27590 consensus_queue.cc:260] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 1.288, Last appended by leader: 288, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:27.353413 27590 raft_consensus.cc:3037] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:27.354323 27590 raft_consensus.cc:2454] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff8b0a555ccd48bcb11dca38d3b2f2f7 in term 2.
I20240501 14:09:27.354696 29177 leader_election.cc:302] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9cc3f1efaf44a4895d3109beabe5312, ff8b0a555ccd48bcb11dca38d3b2f2f7; no voters:
I20240501 14:09:27.354882 30705 raft_consensus.cc:2781] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:27.355095 30705 raft_consensus.cc:686] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 2 LEADER]: Becoming Leader. State: Replica: ff8b0a555ccd48bcb11dca38d3b2f2f7, State: Running, Role: LEADER
I20240501 14:09:27.355391 30705 consensus_queue.cc:237] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 1.288, Last appended by leader: 288, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:27.357537 27447 catalog_manager.cc:5484] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 reported cstate change: term changed from 1 to 2, leader changed from c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) to ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71). New cstate: current_term: 2 leader_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:27.357748 30372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d8c8dbc45b449583f0a1808c634855" candidate_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 288 } } ignore_live_leader: true dest_uuid: "67eb982d9330483183c06df51449cdbd"
I20240501 14:09:27.357910 30372 raft_consensus.cc:3037] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:27.358873 30372 raft_consensus.cc:2454] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff8b0a555ccd48bcb11dca38d3b2f2f7 in term 2.
I20240501 14:09:27.390612 30657 raft_consensus.cc:979] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5: : Instructing follower 67eb982d9330483183c06df51449cdbd to start an election
I20240501 14:09:27.390714 30657 raft_consensus.cc:1067] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Signalling peer 67eb982d9330483183c06df51449cdbd to start an election
I20240501 14:09:27.391173 30372 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a5ec8adfb714bf98dcf3279b9df2148"
dest_uuid: "67eb982d9330483183c06df51449cdbd"
from {username='slave'} at 127.23.191.67:33191
I20240501 14:09:27.391358 30372 raft_consensus.cc:482] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:27.391422 30372 raft_consensus.cc:3037] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:27.392197 30372 raft_consensus.cc:504] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:27.393168 30372 leader_election.cc:288] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [CANDIDATE]: Term 2 election: Requested vote from peers d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647), ff8b0a555ccd48bcb11dca38d3b2f2f7 (127.23.191.71:33963)
I20240501 14:09:27.393455 27853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5ec8adfb714bf98dcf3279b9df2148" candidate_uuid: "67eb982d9330483183c06df51449cdbd" candidate_term: 2 candidate_status { last_received { term: 1 index: 253 } } ignore_live_leader: true dest_uuid: "d1f8294a92e74484ab4a653a4ecf84e5"
I20240501 14:09:27.393566 27853 raft_consensus.cc:3032] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:27.393610 27853 raft_consensus.cc:729] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1f8294a92e74484ab4a653a4ecf84e5, State: Running, Role: LEADER
I20240501 14:09:27.393733 27853 consensus_queue.cc:260] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 1.253, Last appended by leader: 253, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:27.393872 27853 raft_consensus.cc:3037] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:27.394764 27853 raft_consensus.cc:2454] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67eb982d9330483183c06df51449cdbd in term 2.
I20240501 14:09:27.395105 30308 leader_election.cc:302] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67eb982d9330483183c06df51449cdbd, d1f8294a92e74484ab4a653a4ecf84e5; no voters:
I20240501 14:09:27.395311 30627 raft_consensus.cc:2781] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:27.395500 30627 raft_consensus.cc:686] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [term 2 LEADER]: Becoming Leader. State: Replica: 67eb982d9330483183c06df51449cdbd, State: Running, Role: LEADER
I20240501 14:09:27.395787 30627 consensus_queue.cc:237] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 1.253, Last appended by leader: 253, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:27.397123 29241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5ec8adfb714bf98dcf3279b9df2148" candidate_uuid: "67eb982d9330483183c06df51449cdbd" candidate_term: 2 candidate_status { last_received { term: 1 index: 253 } } ignore_live_leader: true dest_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7"
I20240501 14:09:27.397250 29241 raft_consensus.cc:3037] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:27.398156 29241 raft_consensus.cc:2454] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67eb982d9330483183c06df51449cdbd in term 2.
I20240501 14:09:27.398340 27447 catalog_manager.cc:5484] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd reported cstate change: term changed from 1 to 2, leader changed from d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) to 67eb982d9330483183c06df51449cdbd (127.23.191.72). New cstate: current_term: 2 leader_uuid: "67eb982d9330483183c06df51449cdbd" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:27.537708 27718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "085c3852d1ee47dba35aa465b3e36337"
dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44976
I20240501 14:09:27.537840 27718 raft_consensus.cc:595] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:27.611234 30679 raft_consensus.cc:979] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170: : Instructing follower 67eb982d9330483183c06df51449cdbd to start an election
I20240501 14:09:27.611363 30679 raft_consensus.cc:1067] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Signalling peer 67eb982d9330483183c06df51449cdbd to start an election
I20240501 14:09:27.611819 30372 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "085c3852d1ee47dba35aa465b3e36337"
dest_uuid: "67eb982d9330483183c06df51449cdbd"
from {username='slave'} at 127.23.191.66:38117
I20240501 14:09:27.611955 30372 raft_consensus.cc:482] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:27.612015 30372 raft_consensus.cc:3037] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:27.612864 30372 raft_consensus.cc:504] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:27.613374 30372 leader_election.cc:288] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [CANDIDATE]: Term 2 election: Requested vote from peers c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669), fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265)
I20240501 14:09:27.613737 27590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085c3852d1ee47dba35aa465b3e36337" candidate_uuid: "67eb982d9330483183c06df51449cdbd" candidate_term: 2 candidate_status { last_received { term: 1 index: 268 } } ignore_live_leader: true dest_uuid: "c9cc3f1efaf44a4895d3109beabe5312"
I20240501 14:09:27.613751 27718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "085c3852d1ee47dba35aa465b3e36337" candidate_uuid: "67eb982d9330483183c06df51449cdbd" candidate_term: 2 candidate_status { last_received { term: 1 index: 268 } } ignore_live_leader: true dest_uuid: "fec89cd1e3ba491eab76811aae4af170"
I20240501 14:09:27.613858 27718 raft_consensus.cc:3032] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:27.613863 27590 raft_consensus.cc:3037] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:27.613906 27718 raft_consensus.cc:729] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fec89cd1e3ba491eab76811aae4af170, State: Running, Role: LEADER
I20240501 14:09:27.614051 27718 consensus_queue.cc:260] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 1.268, Last appended by leader: 268, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:27.614195 27718 raft_consensus.cc:3037] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:27.614792 27590 raft_consensus.cc:2454] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67eb982d9330483183c06df51449cdbd in term 2.
I20240501 14:09:27.614919 27718 raft_consensus.cc:2454] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67eb982d9330483183c06df51449cdbd in term 2.
I20240501 14:09:27.615105 30305 leader_election.cc:302] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67eb982d9330483183c06df51449cdbd, c9cc3f1efaf44a4895d3109beabe5312; no voters:
I20240501 14:09:27.615307 30822 raft_consensus.cc:2781] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:27.615396 30822 raft_consensus.cc:686] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [term 2 LEADER]: Becoming Leader. State: Replica: 67eb982d9330483183c06df51449cdbd, State: Running, Role: LEADER
I20240501 14:09:27.615526 30822 consensus_queue.cc:237] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 1.268, Last appended by leader: 268, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:27.617141 27447 catalog_manager.cc:5484] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd reported cstate change: term changed from 1 to 2, leader changed from fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) to 67eb982d9330483183c06df51449cdbd (127.23.191.72). New cstate: current_term: 2 leader_uuid: "67eb982d9330483183c06df51449cdbd" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:27.868853 30372 raft_consensus.cc:1261] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd [term 2 FOLLOWER]: Refusing update from remote peer ff8b0a555ccd48bcb11dca38d3b2f2f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 288. Preceding OpId from leader: term: 2 index: 289. (index mismatch)
I20240501 14:09:27.869436 30859 consensus_queue.cc:1035] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20240501 14:09:27.872303 27590 raft_consensus.cc:1261] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Refusing update from remote peer ff8b0a555ccd48bcb11dca38d3b2f2f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 288. Preceding OpId from leader: term: 2 index: 289. (index mismatch)
I20240501 14:09:27.872761 30859 consensus_queue.cc:1035] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20240501 14:09:27.879029 29241 consensus_queue.cc:237] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 289, Committed index: 289, Last appended: 2.289, Last appended by leader: 288, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:27.879987 30372 raft_consensus.cc:1261] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd [term 2 FOLLOWER]: Refusing update from remote peer ff8b0a555ccd48bcb11dca38d3b2f2f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 289. Preceding OpId from leader: term: 2 index: 290. (index mismatch)
I20240501 14:09:27.880338 30859 consensus_queue.cc:1035] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s
I20240501 14:09:27.881803 30860 raft_consensus.cc:2932] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 2 LEADER]: Committing config change with OpId 2.290: config changed from index 288 to 290, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:27.882246 30372 raft_consensus.cc:2932] T 41d8c8dbc45b449583f0a1808c634855 P 67eb982d9330483183c06df51449cdbd [term 2 FOLLOWER]: Committing config change with OpId 2.290: config changed from index 288 to 290, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:27.883598 27436 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 41d8c8dbc45b449583f0a1808c634855 with cas_config_opid_index 288: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:27.884219 27447 catalog_manager.cc:5484] T 41d8c8dbc45b449583f0a1808c634855 P ff8b0a555ccd48bcb11dca38d3b2f2f7 reported cstate change: config changed from index 288 to 290, VOTER c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65) evicted. New cstate: current_term: 2 leader_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" committed_config { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:27.886400 27570 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d8c8dbc45b449583f0a1808c634855 with delete_type TABLET_DATA_TOMBSTONED (TS c9cc3f1efaf44a4895d3109beabe5312 not found in new config with opid_index 290) from {username='slave'} at 127.0.0.1:58578
I20240501 14:09:27.886749 30866 tablet_replica.cc:330] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312: stopping tablet replica
I20240501 14:09:27.886888 30866 raft_consensus.cc:2229] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:27.886993 30866 raft_consensus.cc:2258] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:27.888248 30866 ts_tablet_manager.cc:1875] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:27.890003 30866 ts_tablet_manager.cc:1888] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.289
I20240501 14:09:27.890066 30866 log.cc:1196] T 41d8c8dbc45b449583f0a1808c634855 P c9cc3f1efaf44a4895d3109beabe5312: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/41d8c8dbc45b449583f0a1808c634855
I20240501 14:09:27.890537 27433 catalog_manager.cc:4830] TS c9cc3f1efaf44a4895d3109beabe5312 (127.23.191.65:34669): tablet 41d8c8dbc45b449583f0a1808c634855 (table rebalance_test_table_3 [id=78f3b25b930c48d7bc71a12bc14a9aaa]) successfully deleted
I20240501 14:09:27.913528 29241 raft_consensus.cc:1261] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 2 FOLLOWER]: Refusing update from remote peer 67eb982d9330483183c06df51449cdbd: Log matching property violated. Preceding OpId in replica: term: 1 index: 253. Preceding OpId from leader: term: 2 index: 254. (index mismatch)
I20240501 14:09:27.914002 30822 consensus_queue.cc:1035] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20240501 14:09:27.916389 27853 raft_consensus.cc:1261] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 2 FOLLOWER]: Refusing update from remote peer 67eb982d9330483183c06df51449cdbd: Log matching property violated. Preceding OpId in replica: term: 1 index: 253. Preceding OpId from leader: term: 2 index: 254. (index mismatch)
I20240501 14:09:27.916916 30627 consensus_queue.cc:1035] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f8294a92e74484ab4a653a4ecf84e5" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 45647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20240501 14:09:27.922058 30372 consensus_queue.cc:237] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 2.254, Last appended by leader: 253, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:27.923157 29241 raft_consensus.cc:1261] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 2 FOLLOWER]: Refusing update from remote peer 67eb982d9330483183c06df51449cdbd: Log matching property violated. Preceding OpId in replica: term: 2 index: 254. Preceding OpId from leader: term: 2 index: 255. (index mismatch)
I20240501 14:09:27.923502 30627 consensus_queue.cc:1035] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s
I20240501 14:09:27.924887 30822 raft_consensus.cc:2932] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd [term 2 LEADER]: Committing config change with OpId 2.255: config changed from index 253 to 255, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:27.925272 29241 raft_consensus.cc:2932] T 8a5ec8adfb714bf98dcf3279b9df2148 P ff8b0a555ccd48bcb11dca38d3b2f2f7 [term 2 FOLLOWER]: Committing config change with OpId 2.255: config changed from index 253 to 255, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:27.926239 27434 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a5ec8adfb714bf98dcf3279b9df2148 with cas_config_opid_index 253: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:27.927071 27447 catalog_manager.cc:5484] T 8a5ec8adfb714bf98dcf3279b9df2148 P 67eb982d9330483183c06df51449cdbd reported cstate change: config changed from index 253 to 255, VOTER d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67) evicted. New cstate: current_term: 2 leader_uuid: "67eb982d9330483183c06df51449cdbd" committed_config { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ff8b0a555ccd48bcb11dca38d3b2f2f7" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:27.929275 27832 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a5ec8adfb714bf98dcf3279b9df2148 with delete_type TABLET_DATA_TOMBSTONED (TS d1f8294a92e74484ab4a653a4ecf84e5 not found in new config with opid_index 255) from {username='slave'} at 127.0.0.1:42372
I20240501 14:09:27.929646 30868 tablet_replica.cc:330] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5: stopping tablet replica
I20240501 14:09:27.929777 30868 raft_consensus.cc:2229] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:27.929889 30868 raft_consensus.cc:2258] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:27.931174 30868 ts_tablet_manager.cc:1875] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:27.933089 30868 ts_tablet_manager.cc:1888] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.254
I20240501 14:09:27.933153 30868 log.cc:1196] T 8a5ec8adfb714bf98dcf3279b9df2148 P d1f8294a92e74484ab4a653a4ecf84e5: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/8a5ec8adfb714bf98dcf3279b9df2148
I20240501 14:09:27.933593 27436 catalog_manager.cc:4830] TS d1f8294a92e74484ab4a653a4ecf84e5 (127.23.191.67:45647): tablet 8a5ec8adfb714bf98dcf3279b9df2148 (table rebalance_test_table_2 [id=8615b34ff7ef42beb6766dea8a59f98a]) successfully deleted
I20240501 14:09:28.027606 27590 raft_consensus.cc:1261] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Refusing update from remote peer 67eb982d9330483183c06df51449cdbd: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 2 index: 269. (index mismatch)
I20240501 14:09:28.028048 30627 consensus_queue.cc:1035] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240501 14:09:28.030508 27718 raft_consensus.cc:1261] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 2 FOLLOWER]: Refusing update from remote peer 67eb982d9330483183c06df51449cdbd: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 2 index: 269. (index mismatch)
I20240501 14:09:28.030957 30889 consensus_queue.cc:1035] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec89cd1e3ba491eab76811aae4af170" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 44265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240501 14:09:28.032867 30372 consensus_queue.cc:237] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 269, Committed index: 269, Last appended: 2.269, Last appended by leader: 268, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } }
I20240501 14:09:28.033753 27590 raft_consensus.cc:1261] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Refusing update from remote peer 67eb982d9330483183c06df51449cdbd: Log matching property violated. Preceding OpId in replica: term: 2 index: 269. Preceding OpId from leader: term: 2 index: 270. (index mismatch)
I20240501 14:09:28.034046 30889 consensus_queue.cc:1035] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20240501 14:09:28.035369 30627 raft_consensus.cc:2932] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd [term 2 LEADER]: Committing config change with OpId 2.270: config changed from index 268 to 270, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:28.036653 27434 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 085c3852d1ee47dba35aa465b3e36337 with cas_config_opid_index 268: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:28.037106 27590 raft_consensus.cc:2932] T 085c3852d1ee47dba35aa465b3e36337 P c9cc3f1efaf44a4895d3109beabe5312 [term 2 FOLLOWER]: Committing config change with OpId 2.270: config changed from index 268 to 270, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } } }
I20240501 14:09:28.037596 27447 catalog_manager.cc:5484] T 085c3852d1ee47dba35aa465b3e36337 P 67eb982d9330483183c06df51449cdbd reported cstate change: config changed from index 268 to 270, VOTER fec89cd1e3ba491eab76811aae4af170 (127.23.191.66) evicted. New cstate: current_term: 2 leader_uuid: "67eb982d9330483183c06df51449cdbd" committed_config { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "c9cc3f1efaf44a4895d3109beabe5312" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 34669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67eb982d9330483183c06df51449cdbd" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 36075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:28.039933 27702 tablet_service.cc:1514] Processing DeleteTablet for tablet 085c3852d1ee47dba35aa465b3e36337 with delete_type TABLET_DATA_TOMBSTONED (TS fec89cd1e3ba491eab76811aae4af170 not found in new config with opid_index 270) from {username='slave'} at 127.0.0.1:48430
I20240501 14:09:28.040257 30891 tablet_replica.cc:330] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170: stopping tablet replica
I20240501 14:09:28.040366 30891 raft_consensus.cc:2229] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:28.040450 30891 raft_consensus.cc:2258] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:28.041565 30891 ts_tablet_manager.cc:1875] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:28.043505 30891 ts_tablet_manager.cc:1888] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.269
I20240501 14:09:28.043569 30891 log.cc:1196] T 085c3852d1ee47dba35aa465b3e36337 P fec89cd1e3ba491eab76811aae4af170: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/085c3852d1ee47dba35aa465b3e36337
I20240501 14:09:28.043998 27434 catalog_manager.cc:4830] TS fec89cd1e3ba491eab76811aae4af170 (127.23.191.66:44265): tablet 085c3852d1ee47dba35aa465b3e36337 (table rebalance_test_table_4 [id=7230a83999e24b5fb8afb43b5c8944e4]) successfully deleted
I20240501 14:09:28.259238 30352 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:28.278602 27570 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:28.278820 28826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:28.279526 29091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:28.280246 27832 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:28.283675 27702 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:28.298501 29224 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:28.309548 28959 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:28.386893 28826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:28.388237 29091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:28.392874 27570 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:28.394366 30352 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:28.402240 28959 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:28.403264 27702 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:28.415839 27832 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:28.418180 29224 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:28.514940 28826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:28.520200 28959 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:28.520978 30352 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:28.529085 29091 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:28.529943 29224 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:28.530838 27702 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:28.536957 27570 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:28.542914 27832 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
25f42fba200742378c49fb096885390d | 127.23.191.126:38485 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.191.84:38191 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
3625e18e2ea14d43aa3b86b014fd89f5 | 127.23.191.70:33907 | HEALTHY | <none> | 1 | 0
4c37ce2bb5d844c09969e644d463cf17 | 127.23.191.68:43375 | HEALTHY | <none> | 1 | 0
67eb982d9330483183c06df51449cdbd | 127.23.191.72:36075 | HEALTHY | <none> | 2 | 0
ae4fd219bcae42e3969310879588dc32 | 127.23.191.69:39239 | HEALTHY | <none> | 0 | 0
c9cc3f1efaf44a4895d3109beabe5312 | 127.23.191.65:34669 | HEALTHY | <none> | 3 | 0
d1f8294a92e74484ab4a653a4ecf84e5 | 127.23.191.67:45647 | HEALTHY | <none> | 4 | 0
fec89cd1e3ba491eab76811aae4af170 | 127.23.191.66:44265 | HEALTHY | <none> | 4 | 0
ff8b0a555ccd48bcb11dca38d3b2f2f7 | 127.23.191.71:33963 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.23.191.65 | experimental | 127.23.191.65:34669
local_ip_for_outbound_sockets | 127.23.191.66 | experimental | 127.23.191.66:44265
local_ip_for_outbound_sockets | 127.23.191.67 | experimental | 127.23.191.67:45647
local_ip_for_outbound_sockets | 127.23.191.68 | experimental | 127.23.191.68:43375
local_ip_for_outbound_sockets | 127.23.191.69 | experimental | 127.23.191.69:39239
local_ip_for_outbound_sockets | 127.23.191.70 | experimental | 127.23.191.70:33907
local_ip_for_outbound_sockets | 127.23.191.71 | experimental | 127.23.191.71:33963
local_ip_for_outbound_sockets | 127.23.191.72 | experimental | 127.23.191.72:36075
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.191.65:34669
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.23.191.66:44265
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.191.67:45647
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.23.191.68:43375
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.23.191.69:39239
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.23.191.70:33907
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.23.191.71:33963
server_dump_info_path | /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.23.191.72:36075
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.191.84:38191 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:09:28.574466 24317 log_verifier.cc:126] Checking tablet 085c3852d1ee47dba35aa465b3e36337
I20240501 14:09:28.626752 24317 log_verifier.cc:177] Verified matching terms for 270 ops in tablet 085c3852d1ee47dba35aa465b3e36337
I20240501 14:09:28.626870 24317 log_verifier.cc:126] Checking tablet 322635d2e89b4bd69e1f1442b0f811e8
I20240501 14:09:28.672788 24317 log_verifier.cc:177] Verified matching terms for 273 ops in tablet 322635d2e89b4bd69e1f1442b0f811e8
I20240501 14:09:28.672930 24317 log_verifier.cc:126] Checking tablet 41d8c8dbc45b449583f0a1808c634855
I20240501 14:09:28.723453 24317 log_verifier.cc:177] Verified matching terms for 290 ops in tablet 41d8c8dbc45b449583f0a1808c634855
I20240501 14:09:28.723584 24317 log_verifier.cc:126] Checking tablet 78612cef5af34df3b33567744725af28
I20240501 14:09:28.732435 24317 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 78612cef5af34df3b33567744725af28
I20240501 14:09:28.732496 24317 log_verifier.cc:126] Checking tablet 8a5ec8adfb714bf98dcf3279b9df2148
I20240501 14:09:28.779197 24317 log_verifier.cc:177] Verified matching terms for 255 ops in tablet 8a5ec8adfb714bf98dcf3279b9df2148
I20240501 14:09:28.779321 24317 log_verifier.cc:126] Checking tablet 8cf4c3bcfc524e708092cd92d0622684
I20240501 14:09:28.827966 24317 log_verifier.cc:177] Verified matching terms for 295 ops in tablet 8cf4c3bcfc524e708092cd92d0622684
I20240501 14:09:28.828086 24317 log_verifier.cc:126] Checking tablet ad72089b2cf54cd0b61ee8bb2c95975f
I20240501 14:09:28.875284 24317 log_verifier.cc:177] Verified matching terms for 246 ops in tablet ad72089b2cf54cd0b61ee8bb2c95975f
I20240501 14:09:28.875430 24317 log_verifier.cc:126] Checking tablet b54ea6caa4f6448790b5456a2e5676c2
I20240501 14:09:28.923193 24317 log_verifier.cc:177] Verified matching terms for 284 ops in tablet b54ea6caa4f6448790b5456a2e5676c2
I20240501 14:09:28.923332 24317 log_verifier.cc:126] Checking tablet c717dce4871e459390c508e275c52b31
I20240501 14:09:28.970777 24317 log_verifier.cc:177] Verified matching terms for 275 ops in tablet c717dce4871e459390c508e275c52b31
I20240501 14:09:28.970911 24317 log_verifier.cc:126] Checking tablet ca6f19b8716843ccaabf666651584928
I20240501 14:09:29.020308 24317 log_verifier.cc:177] Verified matching terms for 268 ops in tablet ca6f19b8716843ccaabf666651584928
I20240501 14:09:29.020437 24317 log_verifier.cc:126] Checking tablet dcbe4acb729e48bb89d5a26cb211b2eb
I20240501 14:09:29.071799 24317 log_verifier.cc:177] Verified matching terms for 306 ops in tablet dcbe4acb729e48bb89d5a26cb211b2eb
I20240501 14:09:29.071933 24317 log_verifier.cc:126] Checking tablet e06d6b042b604fd19f19c44c3161beb8
I20240501 14:09:29.119081 24317 log_verifier.cc:177] Verified matching terms for 284 ops in tablet e06d6b042b604fd19f19c44c3161beb8
I20240501 14:09:29.119211 24317 log_verifier.cc:126] Checking tablet e0d87c63b9f64212a3b451bb4140979e
I20240501 14:09:29.173342 24317 log_verifier.cc:177] Verified matching terms for 321 ops in tablet e0d87c63b9f64212a3b451bb4140979e
I20240501 14:09:29.173482 24317 log_verifier.cc:126] Checking tablet ea2e1d9ab9a54d52b89c72c4ef442ce5
I20240501 14:09:29.217443 24317 log_verifier.cc:177] Verified matching terms for 259 ops in tablet ea2e1d9ab9a54d52b89c72c4ef442ce5
I20240501 14:09:29.217589 24317 log_verifier.cc:126] Checking tablet fd80131af5484a938b410e65b6ea73c7
I20240501 14:09:29.270897 24317 log_verifier.cc:177] Verified matching terms for 300 ops in tablet fd80131af5484a938b410e65b6ea73c7
I20240501 14:09:29.271018 24317 log_verifier.cc:126] Checking tablet fe7fe0171f2d477c91661a766239b9fa
I20240501 14:09:29.320242 24317 log_verifier.cc:177] Verified matching terms for 247 ops in tablet fe7fe0171f2d477c91661a766239b9fa
I20240501 14:09:29.321978 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 27509
I20240501 14:09:29.331300 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 27640
I20240501 14:09:29.339286 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 27771
I20240501 14:09:29.347210 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 28763
I20240501 14:09:29.353147 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 28896
I20240501 14:09:29.358461 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 29028
I20240501 14:09:29.363090 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 29161
I20240501 14:09:29.367990 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 30289
I20240501 14:09:29.372758 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 27418
2024-05-01T14:09:29Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (22740 ms)
[----------] 1 test from RebalanceParamTest (22740 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240501 14:09:29.418541 24317 test_util.cc:268] Using random seed: 150018855
I20240501 14:09:29.419291 24317 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:09:29.419329 24317 ts_itest-base.cc:113] --------------
I20240501 14:09:29.419339 24317 ts_itest-base.cc:114] 8 tablet servers
I20240501 14:09:29.419348 24317 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:09:29.419358 24317 ts_itest-base.cc:116] --------------
2024-05-01T14:09:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:09:29Z Disabled control of system clock
I20240501 14:09:29.427110 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.191.126:41449
--webserver_interface=127.23.191.126
--webserver_port=0
--builtin_ntp_servers=127.23.191.84:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.191.126:41449
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 14:09:29.529911 31016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:29.530130 31016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:29.530169 31016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:29.533139 31016 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:29.533205 31016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:09:29.533231 31016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:29.533255 31016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:09:29.533272 31016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:09:29.536736 31016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.23.191.126:41449
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.191.126:41449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.23.191.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.31016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:29.537565 31016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:29.538595 31016 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:29.545660 31024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:29.546013 31022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:29.546646 31021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:29.547165 31016 server_base.cc:1034] running on GCE node
I20240501 14:09:29.547628 31016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:29.548480 31016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:29.549623 31016 hybrid_clock.cc:648] HybridClock initialized: now 1714572569549609 us; error 25 us; skew 500 ppm
I20240501 14:09:29.551405 31016 webserver.cc:460] Webserver started at http://127.23.191.126:46341/ using document root <none> and password file <none>
I20240501 14:09:29.551930 31016 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:29.551985 31016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:29.552148 31016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:29.553778 31016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d913062e66274883a568f59820e333ee"
format_stamp: "Formatted at 2024-05-01 14:09:29 on dist-test-slave-c4g3"
I20240501 14:09:29.554214 31016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d913062e66274883a568f59820e333ee"
format_stamp: "Formatted at 2024-05-01 14:09:29 on dist-test-slave-c4g3"
I20240501 14:09:29.557319 31016 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:29.559357 31030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:29.560276 31016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:29.560381 31016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d913062e66274883a568f59820e333ee"
format_stamp: "Formatted at 2024-05-01 14:09:29 on dist-test-slave-c4g3"
I20240501 14:09:29.560459 31016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:29.578013 31016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:29.578653 31016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:29.578809 31016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:29.585239 31016 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.126:41449
I20240501 14:09:29.585245 31082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.126:41449 every 8 connection(s)
I20240501 14:09:29.586163 31016 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:09:29.588587 31083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.591936 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 31016
I20240501 14:09:29.592077 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:09:29.593556 31083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee: Bootstrap starting.
I20240501 14:09:29.596024 31083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.596858 31083 log.cc:824] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:29.598611 31083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee: No bootstrap required, opened a new log
I20240501 14:09:29.601531 31083 raft_consensus.cc:348] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d913062e66274883a568f59820e333ee" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 41449 } }
I20240501 14:09:29.601701 31083 raft_consensus.cc:374] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.601745 31083 raft_consensus.cc:729] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d913062e66274883a568f59820e333ee, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.602088 31083 consensus_queue.cc:260] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d913062e66274883a568f59820e333ee" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 41449 } }
I20240501 14:09:29.602186 31083 raft_consensus.cc:388] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:29.602227 31083 raft_consensus.cc:482] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:29.602301 31083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.602968 31083 raft_consensus.cc:504] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d913062e66274883a568f59820e333ee" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 41449 } }
I20240501 14:09:29.603258 31083 leader_election.cc:302] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d913062e66274883a568f59820e333ee; no voters:
I20240501 14:09:29.603472 31083 leader_election.cc:288] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:29.603564 31088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:29.604162 31088 raft_consensus.cc:686] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [term 1 LEADER]: Becoming Leader. State: Replica: d913062e66274883a568f59820e333ee, State: Running, Role: LEADER
I20240501 14:09:29.604463 31088 consensus_queue.cc:237] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d913062e66274883a568f59820e333ee" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 41449 } }
I20240501 14:09:29.604542 31083 sys_catalog.cc:564] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:09:29.605760 31090 sys_catalog.cc:455] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader d913062e66274883a568f59820e333ee. Latest consensus state: current_term: 1 leader_uuid: "d913062e66274883a568f59820e333ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d913062e66274883a568f59820e333ee" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 41449 } } }
I20240501 14:09:29.605854 31090 sys_catalog.cc:458] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:29.606169 31097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:09:29.606120 31089 sys_catalog.cc:455] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d913062e66274883a568f59820e333ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d913062e66274883a568f59820e333ee" member_type: VOTER last_known_addr { host: "127.23.191.126" port: 41449 } } }
I20240501 14:09:29.606233 31089 sys_catalog.cc:458] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:29.609037 31097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:09:29.613837 31097 catalog_manager.cc:1347] Generated new cluster ID: 77a17a41f6324ab382a521777bdda212
I20240501 14:09:29.613890 31097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:09:29.620954 31097 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:09:29.621871 31097 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:09:29.628434 31097 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d913062e66274883a568f59820e333ee: Generated new TSK 0
I20240501 14:09:29.629070 31097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:09:29.638167 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.65:0
--local_ip_for_outbound_sockets=127.23.191.65
--webserver_interface=127.23.191.65
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--builtin_ntp_servers=127.23.191.84:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:29.754909 31107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:29.755137 31107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:29.755182 31107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:29.758272 31107 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:29.758340 31107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:29.758415 31107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.65
I20240501 14:09:29.765152 31107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.23.191.65
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--never_fsync=true
--heap_profile_path=/tmp/kudu.31107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.65
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:29.766077 31107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:29.767238 31107 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:29.775640 31115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:29.776157 31112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:29.776610 31113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:29.777235 31107 server_base.cc:1034] running on GCE node
I20240501 14:09:29.777583 31107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:29.778126 31107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:29.779273 31107 hybrid_clock.cc:648] HybridClock initialized: now 1714572569779256 us; error 29 us; skew 500 ppm
I20240501 14:09:29.781126 31107 webserver.cc:460] Webserver started at http://127.23.191.65:40713/ using document root <none> and password file <none>
I20240501 14:09:29.781651 31107 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:29.781708 31107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:29.781868 31107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:29.783460 31107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0e4622d80cc94f12b492bdd7597a61e1"
format_stamp: "Formatted at 2024-05-01 14:09:29 on dist-test-slave-c4g3"
I20240501 14:09:29.783910 31107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0e4622d80cc94f12b492bdd7597a61e1"
format_stamp: "Formatted at 2024-05-01 14:09:29 on dist-test-slave-c4g3"
I20240501 14:09:29.787009 31107 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 14:09:29.789016 31121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:29.789928 31107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:29.790046 31107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0e4622d80cc94f12b492bdd7597a61e1"
format_stamp: "Formatted at 2024-05-01 14:09:29 on dist-test-slave-c4g3"
I20240501 14:09:29.790125 31107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:29.817302 31107 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:29.817950 31107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:29.818158 31107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:29.818717 31107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:29.819633 31107 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:29.819690 31107 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:29.819742 31107 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:29.819766 31107 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:29.827589 31107 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.65:32841
I20240501 14:09:29.827636 31234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.65:32841 every 8 connection(s)
I20240501 14:09:29.828469 31107 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:09:29.834311 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 31107
I20240501 14:09:29.834451 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:09:29.836478 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.66:0
--local_ip_for_outbound_sockets=127.23.191.66
--webserver_interface=127.23.191.66
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--builtin_ntp_servers=127.23.191.84:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:29.836863 31235 heartbeater.cc:347] Connected to a master server at 127.23.191.126:41449
I20240501 14:09:29.837090 31235 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:29.837668 31235 heartbeater.cc:511] Master 127.23.191.126:41449 requested a full tablet report, sending...
I20240501 14:09:29.839184 31047 ts_manager.cc:194] Registered new tserver with Master: 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:29.840732 31047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.65:45535
W20240501 14:09:29.939462 31238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:29.939697 31238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:29.939741 31238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:29.942747 31238 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:29.942812 31238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:29.942876 31238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.66
I20240501 14:09:29.946398 31238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.23.191.66
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--never_fsync=true
--heap_profile_path=/tmp/kudu.31238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.66
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:29.947252 31238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:29.948293 31238 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:29.956398 31243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:29.956698 31246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:29.957379 31244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:29.957885 31238 server_base.cc:1034] running on GCE node
I20240501 14:09:29.958232 31238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:29.958752 31238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:29.959894 31238 hybrid_clock.cc:648] HybridClock initialized: now 1714572569959864 us; error 46 us; skew 500 ppm
I20240501 14:09:29.961738 31238 webserver.cc:460] Webserver started at http://127.23.191.66:40613/ using document root <none> and password file <none>
I20240501 14:09:29.962234 31238 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:29.962282 31238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:29.962436 31238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:29.964054 31238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
format_stamp: "Formatted at 2024-05-01 14:09:29 on dist-test-slave-c4g3"
I20240501 14:09:29.964464 31238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
format_stamp: "Formatted at 2024-05-01 14:09:29 on dist-test-slave-c4g3"
I20240501 14:09:29.967633 31238 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:29.969950 31252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:29.970955 31238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:29.971076 31238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
format_stamp: "Formatted at 2024-05-01 14:09:29 on dist-test-slave-c4g3"
I20240501 14:09:29.971165 31238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:30.002601 31238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:30.003237 31238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:30.003391 31238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:30.003914 31238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:30.004801 31238 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:30.004885 31238 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.004951 31238 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:30.005004 31238 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.013119 31238 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.66:45265
I20240501 14:09:30.013150 31365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.66:45265 every 8 connection(s)
I20240501 14:09:30.014043 31238 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:09:30.022220 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 31238
I20240501 14:09:30.022302 31366 heartbeater.cc:347] Connected to a master server at 127.23.191.126:41449
I20240501 14:09:30.022346 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:09:30.022527 31366 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:30.023129 31366 heartbeater.cc:511] Master 127.23.191.126:41449 requested a full tablet report, sending...
I20240501 14:09:30.024114 31047 ts_manager.cc:194] Registered new tserver with Master: 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:30.024494 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.67:0
--local_ip_for_outbound_sockets=127.23.191.67
--webserver_interface=127.23.191.67
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--builtin_ntp_servers=127.23.191.84:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:30.024811 31047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.66:60535
W20240501 14:09:30.128509 31369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:30.128744 31369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:30.128804 31369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:30.131748 31369 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:30.131816 31369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:30.131901 31369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.67
I20240501 14:09:30.135375 31369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.23.191.67
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--never_fsync=true
--heap_profile_path=/tmp/kudu.31369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.67
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:30.136255 31369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:30.137363 31369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:30.145604 31374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:30.146162 31375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:30.146605 31377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:30.147264 31369 server_base.cc:1034] running on GCE node
I20240501 14:09:30.147661 31369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:30.148192 31369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:30.149328 31369 hybrid_clock.cc:648] HybridClock initialized: now 1714572570149311 us; error 31 us; skew 500 ppm
I20240501 14:09:30.151136 31369 webserver.cc:460] Webserver started at http://127.23.191.67:42747/ using document root <none> and password file <none>
I20240501 14:09:30.151671 31369 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:30.151724 31369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:30.151937 31369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:30.153730 31369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6a6fcc099d1340c383e5a429643d5b7c"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.154186 31369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6a6fcc099d1340c383e5a429643d5b7c"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.157382 31369 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:30.159504 31383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.160467 31369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:30.160575 31369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6a6fcc099d1340c383e5a429643d5b7c"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.160663 31369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:30.176693 31369 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:30.177301 31369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:30.177453 31369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:30.177973 31369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:30.178848 31369 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:30.178894 31369 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.178937 31369 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:30.178965 31369 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.187027 31369 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.67:39413
I20240501 14:09:30.187050 31496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.67:39413 every 8 connection(s)
I20240501 14:09:30.187942 31369 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:09:30.189627 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 31369
I20240501 14:09:30.189754 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:09:30.191915 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.68:0
--local_ip_for_outbound_sockets=127.23.191.68
--webserver_interface=127.23.191.68
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--builtin_ntp_servers=127.23.191.84:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:30.197367 31497 heartbeater.cc:347] Connected to a master server at 127.23.191.126:41449
I20240501 14:09:30.197588 31497 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:30.198083 31497 heartbeater.cc:511] Master 127.23.191.126:41449 requested a full tablet report, sending...
I20240501 14:09:30.199023 31047 ts_manager.cc:194] Registered new tserver with Master: 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:30.199545 31047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.67:50447
W20240501 14:09:30.296267 31500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:30.296514 31500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:30.296583 31500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:30.299666 31500 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:30.299744 31500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:30.299832 31500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.68
I20240501 14:09:30.303521 31500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.23.191.68
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--never_fsync=true
--heap_profile_path=/tmp/kudu.31500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.68
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:30.304425 31500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:30.305635 31500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:30.313721 31505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:30.314582 31506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:30.315138 31508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:30.315692 31500 server_base.cc:1034] running on GCE node
I20240501 14:09:30.316073 31500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:30.316628 31500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:30.317771 31500 hybrid_clock.cc:648] HybridClock initialized: now 1714572570317748 us; error 39 us; skew 500 ppm
I20240501 14:09:30.319736 31500 webserver.cc:460] Webserver started at http://127.23.191.68:34317/ using document root <none> and password file <none>
I20240501 14:09:30.320255 31500 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:30.320304 31500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:30.320459 31500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:30.322062 31500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.322480 31500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.325707 31500 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:30.327775 31514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.328739 31500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:30.328855 31500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.328943 31500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:30.340616 31500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:30.341199 31500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:30.341334 31500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:30.341804 31500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:30.342643 31500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:30.342686 31500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.342736 31500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:30.342763 31500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.350697 31500 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.68:45349
I20240501 14:09:30.350724 31627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.68:45349 every 8 connection(s)
I20240501 14:09:30.351588 31500 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:09:30.357551 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 31500
I20240501 14:09:30.357659 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:09:30.359685 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.69:0
--local_ip_for_outbound_sockets=127.23.191.69
--webserver_interface=127.23.191.69
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--builtin_ntp_servers=127.23.191.84:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:30.359896 31628 heartbeater.cc:347] Connected to a master server at 127.23.191.126:41449
I20240501 14:09:30.360107 31628 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:30.360580 31628 heartbeater.cc:511] Master 127.23.191.126:41449 requested a full tablet report, sending...
I20240501 14:09:30.361526 31047 ts_manager.cc:194] Registered new tserver with Master: e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:30.362025 31047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.68:40711
W20240501 14:09:30.463093 31631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:30.463330 31631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:30.463376 31631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:30.466502 31631 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:30.466580 31631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:30.466665 31631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.69
I20240501 14:09:30.471189 31631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.23.191.69
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--never_fsync=true
--heap_profile_path=/tmp/kudu.31631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.69
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:30.472103 31631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:30.473243 31631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:30.481427 31637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:30.481770 31636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:30.482481 31639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:30.482946 31631 server_base.cc:1034] running on GCE node
I20240501 14:09:30.483321 31631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:30.483896 31631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:30.485030 31631 hybrid_clock.cc:648] HybridClock initialized: now 1714572570485011 us; error 40 us; skew 500 ppm
I20240501 14:09:30.486871 31631 webserver.cc:460] Webserver started at http://127.23.191.69:44577/ using document root <none> and password file <none>
I20240501 14:09:30.487417 31631 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:30.487476 31631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:30.487638 31631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:30.489277 31631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b934fd09590a40ada094b9715d990a00"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.489707 31631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b934fd09590a40ada094b9715d990a00"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.492808 31631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:30.494947 31645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.495916 31631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:30.496017 31631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b934fd09590a40ada094b9715d990a00"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.496107 31631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:30.513620 31631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:30.514202 31631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:30.514350 31631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:30.514853 31631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:30.515805 31631 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:30.515861 31631 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.515928 31631 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:30.516033 31631 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.524596 31631 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.69:45429
I20240501 14:09:30.524618 31758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.69:45429 every 8 connection(s)
I20240501 14:09:30.525537 31631 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:09:30.525810 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 31631
I20240501 14:09:30.525918 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:09:30.528331 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.70:0
--local_ip_for_outbound_sockets=127.23.191.70
--webserver_interface=127.23.191.70
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--builtin_ntp_servers=127.23.191.84:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:30.535039 31759 heartbeater.cc:347] Connected to a master server at 127.23.191.126:41449
I20240501 14:09:30.535288 31759 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:30.535945 31759 heartbeater.cc:511] Master 127.23.191.126:41449 requested a full tablet report, sending...
I20240501 14:09:30.537006 31047 ts_manager.cc:194] Registered new tserver with Master: b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429)
I20240501 14:09:30.537544 31047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.69:55815
W20240501 14:09:30.636067 31762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:30.636301 31762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:30.636351 31762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:30.639397 31762 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:30.639472 31762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:30.639567 31762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.70
I20240501 14:09:30.643206 31762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.23.191.70
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--never_fsync=true
--heap_profile_path=/tmp/kudu.31762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.70
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:30.644086 31762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:30.645215 31762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:30.653254 31770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:30.653510 31768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:30.654136 31767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:30.654688 31762 server_base.cc:1034] running on GCE node
I20240501 14:09:30.655052 31762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:30.655591 31762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:30.656746 31762 hybrid_clock.cc:648] HybridClock initialized: now 1714572570656724 us; error 34 us; skew 500 ppm
I20240501 14:09:30.658607 31762 webserver.cc:460] Webserver started at http://127.23.191.70:41227/ using document root <none> and password file <none>
I20240501 14:09:30.659134 31762 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:30.659191 31762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:30.659355 31762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:30.663916 31762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9438789dc61f4501ad1d4ceaeccee458"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.664404 31762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9438789dc61f4501ad1d4ceaeccee458"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.667619 31762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:30.669721 31776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.670665 31762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:30.670766 31762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9438789dc61f4501ad1d4ceaeccee458"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.670845 31762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:30.683604 31762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:30.684170 31762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:30.684298 31762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:30.684744 31762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:30.685577 31762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:30.685622 31762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.685660 31762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:30.685680 31762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.693254 31762 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.70:38181
I20240501 14:09:30.693285 31889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.70:38181 every 8 connection(s)
I20240501 14:09:30.694209 31762 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:09:30.702555 31890 heartbeater.cc:347] Connected to a master server at 127.23.191.126:41449
I20240501 14:09:30.702769 31890 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:30.703276 31890 heartbeater.cc:511] Master 127.23.191.126:41449 requested a full tablet report, sending...
I20240501 14:09:30.704174 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 31762
I20240501 14:09:30.704267 31047 ts_manager.cc:194] Registered new tserver with Master: 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:09:30.704299 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:09:30.704981 31047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.70:44683
I20240501 14:09:30.706653 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.71:0
--local_ip_for_outbound_sockets=127.23.191.71
--webserver_interface=127.23.191.71
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--builtin_ntp_servers=127.23.191.84:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:30.811383 31893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:30.811601 31893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:30.811645 31893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:30.814684 31893 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:30.814754 31893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:30.814821 31893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.71
I20240501 14:09:30.818315 31893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.23.191.71
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--never_fsync=true
--heap_profile_path=/tmp/kudu.31893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.71
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:30.819154 31893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:30.820240 31893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:30.828367 31901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:30.828879 31899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:30.829290 31898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:30.829962 31893 server_base.cc:1034] running on GCE node
I20240501 14:09:30.830304 31893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:30.830832 31893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:30.831982 31893 hybrid_clock.cc:648] HybridClock initialized: now 1714572570831953 us; error 41 us; skew 500 ppm
I20240501 14:09:30.833807 31893 webserver.cc:460] Webserver started at http://127.23.191.71:36285/ using document root <none> and password file <none>
I20240501 14:09:30.834334 31893 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:30.834393 31893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:30.834556 31893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:30.836133 31893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "a0caf2974041408bb32362bfc4284eaa"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.836580 31893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "a0caf2974041408bb32362bfc4284eaa"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.839705 31893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:30.841850 31907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.842818 31893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:30.842947 31235 heartbeater.cc:503] Master 127.23.191.126:41449 was elected leader, sending a full tablet report...
I20240501 14:09:30.842938 31893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a0caf2974041408bb32362bfc4284eaa"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:30.843021 31893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:30.866642 31893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:30.867274 31893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:30.867434 31893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:30.867956 31893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:30.868813 31893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:30.868901 31893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.868964 31893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:30.868988 31893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:30.877007 31893 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.71:35407
I20240501 14:09:30.877015 32020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.71:35407 every 8 connection(s)
I20240501 14:09:30.877964 31893 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:09:30.882239 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 31893
I20240501 14:09:30.882376 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:09:30.884681 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.72:0
--local_ip_for_outbound_sockets=127.23.191.72
--webserver_interface=127.23.191.72
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--builtin_ntp_servers=127.23.191.84:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:30.887260 32021 heartbeater.cc:347] Connected to a master server at 127.23.191.126:41449
I20240501 14:09:30.887499 32021 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:30.888044 32021 heartbeater.cc:511] Master 127.23.191.126:41449 requested a full tablet report, sending...
I20240501 14:09:30.889014 31046 ts_manager.cc:194] Registered new tserver with Master: a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
I20240501 14:09:30.889559 31046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.71:42443
W20240501 14:09:30.988909 32024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:30.989133 32024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:30.989193 32024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:30.992177 32024 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:30.992244 32024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:30.992328 32024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.72
I20240501 14:09:30.995918 32024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.23.191.72
--webserver_port=0
--tserver_master_addrs=127.23.191.126:41449
--never_fsync=true
--heap_profile_path=/tmp/kudu.32024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.72
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:30.996765 32024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:30.997917 32024 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:31.005910 32029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:31.006484 32030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:31.006886 32032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:31.007422 32024 server_base.cc:1034] running on GCE node
I20240501 14:09:31.007790 32024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:31.008327 32024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:31.009478 32024 hybrid_clock.cc:648] HybridClock initialized: now 1714572571009466 us; error 25 us; skew 500 ppm
I20240501 14:09:31.011340 32024 webserver.cc:460] Webserver started at http://127.23.191.72:43401/ using document root <none> and password file <none>
I20240501 14:09:31.011874 32024 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:31.011932 32024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:31.012100 32024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:31.013770 32024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "a63e0dd1ba2e4a8db8ace9718a700c32"
format_stamp: "Formatted at 2024-05-01 14:09:31 on dist-test-slave-c4g3"
I20240501 14:09:31.014202 32024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "a63e0dd1ba2e4a8db8ace9718a700c32"
format_stamp: "Formatted at 2024-05-01 14:09:31 on dist-test-slave-c4g3"
I20240501 14:09:31.017336 32024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:31.019405 32038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.020363 32024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:31.020468 32024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "a63e0dd1ba2e4a8db8ace9718a700c32"
format_stamp: "Formatted at 2024-05-01 14:09:31 on dist-test-slave-c4g3"
I20240501 14:09:31.020555 32024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:31.026661 31366 heartbeater.cc:503] Master 127.23.191.126:41449 was elected leader, sending a full tablet report...
I20240501 14:09:31.029865 32024 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:31.030418 32024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:31.030562 32024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:31.031054 32024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:31.031885 32024 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:31.031932 32024 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.031982 32024 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:31.032034 32024 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.039475 32024 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.72:39799
I20240501 14:09:31.039506 32151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.72:39799 every 8 connection(s)
I20240501 14:09:31.040366 32024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:09:31.048705 32152 heartbeater.cc:347] Connected to a master server at 127.23.191.126:41449
I20240501 14:09:31.048985 32152 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:31.049521 32152 heartbeater.cc:511] Master 127.23.191.126:41449 requested a full tablet report, sending...
I20240501 14:09:31.050135 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 32024
I20240501 14:09:31.050257 24317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:09:31.050557 31046 ts_manager.cc:194] Registered new tserver with Master: a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799)
I20240501 14:09:31.051142 31046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.72:60325
I20240501 14:09:31.052536 24317 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:09:31.061617 24317 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:09:31.061697 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 31631
I20240501 14:09:31.067693 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 31762
I20240501 14:09:31.072750 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 31893
I20240501 14:09:31.077294 24317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu with pid 32024
I20240501 14:09:31.201406 31497 heartbeater.cc:503] Master 127.23.191.126:41449 was elected leader, sending a full tablet report...
I20240501 14:09:31.364180 31628 heartbeater.cc:503] Master 127.23.191.126:41449 was elected leader, sending a full tablet report...
I20240501 14:09:34.833360 31046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52370:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:34.852066 31431 tablet_service.cc:1467] Processing CreateTablet for tablet 7a542c07a3f841b0a31051a8a37083bf (DEFAULT_TABLE table=rebalance_test_table_0 [id=fbe6d56762d84933a7b7ddaf8c314b4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.852066 31168 tablet_service.cc:1467] Processing CreateTablet for tablet f26d16499ab54a3e9059f6e4ff041177 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fbe6d56762d84933a7b7ddaf8c314b4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.852932 31430 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2e30cb9de14fe19273b3cef801cb67 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fbe6d56762d84933a7b7ddaf8c314b4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.853034 31431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a542c07a3f841b0a31051a8a37083bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.853034 31168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26d16499ab54a3e9059f6e4ff041177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.853549 31167 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2e30cb9de14fe19273b3cef801cb67 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fbe6d56762d84933a7b7ddaf8c314b4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.853781 31167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2e30cb9de14fe19273b3cef801cb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.854221 31300 tablet_service.cc:1467] Processing CreateTablet for tablet 7a542c07a3f841b0a31051a8a37083bf (DEFAULT_TABLE table=rebalance_test_table_0 [id=fbe6d56762d84933a7b7ddaf8c314b4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.854221 31299 tablet_service.cc:1467] Processing CreateTablet for tablet f26d16499ab54a3e9059f6e4ff041177 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fbe6d56762d84933a7b7ddaf8c314b4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.855080 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26d16499ab54a3e9059f6e4ff041177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.856648 31169 tablet_service.cc:1467] Processing CreateTablet for tablet 7a542c07a3f841b0a31051a8a37083bf (DEFAULT_TABLE table=rebalance_test_table_0 [id=fbe6d56762d84933a7b7ddaf8c314b4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.856899 31430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2e30cb9de14fe19273b3cef801cb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.858625 31562 tablet_service.cc:1467] Processing CreateTablet for tablet f26d16499ab54a3e9059f6e4ff041177 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fbe6d56762d84933a7b7ddaf8c314b4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.859205 31300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a542c07a3f841b0a31051a8a37083bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.859401 31562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26d16499ab54a3e9059f6e4ff041177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.861059 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a542c07a3f841b0a31051a8a37083bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.861224 31561 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2e30cb9de14fe19273b3cef801cb67 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fbe6d56762d84933a7b7ddaf8c314b4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.862346 31561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2e30cb9de14fe19273b3cef801cb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.863157 32172 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:34.863947 32174 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:34.865108 32172 tablet_bootstrap.cc:654] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.865860 32172 log.cc:824] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:34.865916 32174 tablet_bootstrap.cc:654] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.866775 32174 log.cc:824] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:34.867362 32176 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:34.868019 32173 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:34.868388 32174 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:34.868572 32174 ts_tablet_manager.cc:1367] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:09:34.869570 32176 tablet_bootstrap.cc:654] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.869743 32173 tablet_bootstrap.cc:654] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.870260 32172 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:34.870301 32176 log.cc:824] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:34.870436 32173 log.cc:824] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:34.870563 32172 ts_tablet_manager.cc:1367] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 14:09:34.871803 32176 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:34.871960 32176 ts_tablet_manager.cc:1367] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:09:34.872001 32173 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:34.872155 32173 ts_tablet_manager.cc:1367] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:34.872177 32174 raft_consensus.cc:348] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:34.872388 32174 raft_consensus.cc:374] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.872442 32174 raft_consensus.cc:729] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.872946 32174 consensus_queue.cc:260] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:34.873842 32174 ts_tablet_manager.cc:1398] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:09:34.874198 32174 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:34.874879 32172 raft_consensus.cc:348] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:34.875133 32174 tablet_bootstrap.cc:654] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.875150 32172 raft_consensus.cc:374] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.875276 32172 raft_consensus.cc:729] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.875823 32173 raft_consensus.cc:348] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:34.876008 32173 raft_consensus.cc:374] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.876057 32173 raft_consensus.cc:729] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.876220 32174 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:34.876291 32174 ts_tablet_manager.cc:1367] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:34.876412 32173 consensus_queue.cc:260] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:34.876773 32174 raft_consensus.cc:348] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:34.876888 32174 raft_consensus.cc:374] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.876917 32174 raft_consensus.cc:729] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.877034 32174 consensus_queue.cc:260] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:34.877255 32173 ts_tablet_manager.cc:1398] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:09:34.877368 32174 ts_tablet_manager.cc:1398] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.877532 32174 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:34.877718 32172 consensus_queue.cc:260] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:34.878315 32174 tablet_bootstrap.cc:654] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.878376 32173 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:34.878671 32172 ts_tablet_manager.cc:1398] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.008s user 0.000s sys 0.005s
I20240501 14:09:34.879055 32172 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:34.879364 32174 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:34.879449 32174 ts_tablet_manager.cc:1367] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:34.879858 32174 raft_consensus.cc:348] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:34.879958 32174 raft_consensus.cc:374] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.879997 32174 raft_consensus.cc:729] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.880038 32173 tablet_bootstrap.cc:654] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.880118 32174 consensus_queue.cc:260] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:34.880393 32172 tablet_bootstrap.cc:654] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.880455 32174 ts_tablet_manager.cc:1398] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.880539 32176 raft_consensus.cc:348] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:34.881796 32173 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:34.881825 32176 raft_consensus.cc:374] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.881865 32173 ts_tablet_manager.cc:1367] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:34.881873 32176 raft_consensus.cc:729] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.882298 32176 consensus_queue.cc:260] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:34.882359 32173 raft_consensus.cc:348] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:34.882438 32173 raft_consensus.cc:374] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.882464 32173 raft_consensus.cc:729] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.882572 32173 consensus_queue.cc:260] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:34.882808 32173 ts_tablet_manager.cc:1398] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.883136 32176 ts_tablet_manager.cc:1398] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20240501 14:09:34.883461 32176 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:34.884335 32176 tablet_bootstrap.cc:654] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.885152 32172 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:34.885236 32172 ts_tablet_manager.cc:1367] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:34.885684 32176 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:34.885771 32176 ts_tablet_manager.cc:1367] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:34.885721 32172 raft_consensus.cc:348] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:34.885814 32172 raft_consensus.cc:374] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.885852 32172 raft_consensus.cc:729] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.885972 32172 consensus_queue.cc:260] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:34.886263 32172 ts_tablet_manager.cc:1398] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.886251 32176 raft_consensus.cc:348] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:34.886341 32176 raft_consensus.cc:374] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.886720 32176 raft_consensus.cc:729] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.886843 32176 consensus_queue.cc:260] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:34.887094 32176 ts_tablet_manager.cc:1398] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:34.899278 32182 raft_consensus.cc:482] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:34.899452 32182 raft_consensus.cc:504] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:34.900436 32182 leader_election.cc:288] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:34.904088 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26d16499ab54a3e9059f6e4ff041177" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:09:34.904089 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26d16499ab54a3e9059f6e4ff041177" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:34.904345 31582 raft_consensus.cc:2454] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:34.904368 31189 raft_consensus.cc:2454] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:34.904788 31255 leader_election.cc:302] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:34.905058 32182 raft_consensus.cc:2781] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:34.905129 32182 raft_consensus.cc:482] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:34.905176 32182 raft_consensus.cc:3037] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.906118 32182 raft_consensus.cc:504] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:34.906458 32182 leader_election.cc:288] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Requested vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:34.906833 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26d16499ab54a3e9059f6e4ff041177" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:34.906858 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26d16499ab54a3e9059f6e4ff041177" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:34.906956 31582 raft_consensus.cc:3037] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.906987 31189 raft_consensus.cc:3037] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.907868 31582 raft_consensus.cc:2454] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:34.908120 31255 leader_election.cc:302] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:34.908182 31189 raft_consensus.cc:2454] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:34.908280 32182 raft_consensus.cc:2781] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:34.908500 32182 raft_consensus.cc:686] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Becoming Leader. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Running, Role: LEADER
I20240501 14:09:34.908824 32182 consensus_queue.cc:237] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:34.911631 31045 catalog_manager.cc:5484] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66). New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:34.942327 31498 tablet.cc:2369] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:09:35.018257 31367 tablet.cc:2369] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:35.054888 32186 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.055032 32186 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:35.056154 32186 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:35.059746 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e30cb9de14fe19273b3cef801cb67" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:35.059898 31582 raft_consensus.cc:2454] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:35.059949 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e30cb9de14fe19273b3cef801cb67" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:35.060225 31124 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:35.060250 31451 raft_consensus.cc:2454] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:35.060453 32186 raft_consensus.cc:2781] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.060523 32186 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.060560 32186 raft_consensus.cc:3037] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.061432 32186 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:35.061733 32186 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Requested vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:35.062017 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e30cb9de14fe19273b3cef801cb67" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:35.062033 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e30cb9de14fe19273b3cef801cb67" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:35.062115 31582 raft_consensus.cc:3037] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.062114 31451 raft_consensus.cc:3037] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.062805 31582 raft_consensus.cc:2454] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:35.062906 31451 raft_consensus.cc:2454] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:35.063064 31124 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:35.063226 32186 raft_consensus.cc:2781] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.063308 32186 raft_consensus.cc:686] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Becoming Leader. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Running, Role: LEADER
I20240501 14:09:35.063594 32186 consensus_queue.cc:237] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:35.066391 31045 catalog_manager.cc:5484] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:35.082955 31236 tablet.cc:2369] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:09:35.105648 31629 tablet.cc:2369] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:35.137400 32184 raft_consensus.cc:482] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.137544 32184 raft_consensus.cc:504] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:35.138636 32184 leader_election.cc:288] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:35.142199 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a542c07a3f841b0a31051a8a37083bf" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:09:35.142372 31189 raft_consensus.cc:2454] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 0.
I20240501 14:09:35.142377 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a542c07a3f841b0a31051a8a37083bf" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" is_pre_election: true
I20240501 14:09:35.142571 31320 raft_consensus.cc:2454] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 0.
I20240501 14:09:35.142745 31384 leader_election.cc:302] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:35.143055 32184 raft_consensus.cc:2781] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.143124 32184 raft_consensus.cc:482] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.143167 32184 raft_consensus.cc:3037] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.144212 32184 raft_consensus.cc:504] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:35.144546 32184 leader_election.cc:288] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 election: Requested vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:35.144937 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a542c07a3f841b0a31051a8a37083bf" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:35.144934 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a542c07a3f841b0a31051a8a37083bf" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:35.145063 31189 raft_consensus.cc:3037] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.145063 31320 raft_consensus.cc:3037] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.145978 31320 raft_consensus.cc:2454] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 1.
I20240501 14:09:35.145992 31189 raft_consensus.cc:2454] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 1.
I20240501 14:09:35.146288 31387 leader_election.cc:302] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:35.146449 32184 raft_consensus.cc:2781] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.146663 32184 raft_consensus.cc:686] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Becoming Leader. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Running, Role: LEADER
I20240501 14:09:35.146972 32184 consensus_queue.cc:237] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:35.149829 31045 catalog_manager.cc:5484] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67). New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:35.309173 32202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:35.309422 32202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:35.312424 32202 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:09:35.336467 32182 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240501 14:09:35.341074 32182 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.403169 31582 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.403240 31451 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.403887 32183 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.404603 32183 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.415283 32242 mvcc.cc:205] Tried to move back new op lower bound from 7022889268841402368 to 7022889267462930432. Current Snapshot: MvccSnapshot[applied={T|T < 7022889268841402368}]
I20240501 14:09:35.416886 32220 mvcc.cc:205] Tried to move back new op lower bound from 7022889268881092608 to 7022889267462930432. Current Snapshot: MvccSnapshot[applied={T|T < 7022889268841402368}]
I20240501 14:09:35.417282 31320 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.418704 32184 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.419970 32223 mvcc.cc:205] Tried to move back new op lower bound from 7022889268841402368 to 7022889267462930432. Current Snapshot: MvccSnapshot[applied={T|T < 7022889268841402368}]
I20240501 14:09:35.422511 31189 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.422916 32184 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.443799 32217 mvcc.cc:205] Tried to move back new op lower bound from 7022889268895735808 to 7022889267804491776. Current Snapshot: MvccSnapshot[applied={T|T < 7022889268895735808}]
INSERT report
rows total: 2000
time total: 106.967 ms
time per row: 0.0534834 ms
I20240501 14:09:35.469982 31041 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52370:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:35.475226 31299 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2c173b2c32494eb16e46a51ca36e31 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.475266 31430 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2c173b2c32494eb16e46a51ca36e31 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.475564 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2c173b2c32494eb16e46a51ca36e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.475572 31430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2c173b2c32494eb16e46a51ca36e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.476125 31431 tablet_service.cc:1467] Processing CreateTablet for tablet ea66fe3d0da04d18b961927a9c988a20 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.476197 31169 tablet_service.cc:1467] Processing CreateTablet for tablet 0399a419a25c4981a2ee90d76cbf59b1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.476333 31431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea66fe3d0da04d18b961927a9c988a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.476447 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0399a419a25c4981a2ee90d76cbf59b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.478281 32262 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:35.478271 31429 tablet_service.cc:1467] Processing CreateTablet for tablet 0399a419a25c4981a2ee90d76cbf59b1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.478436 31299 tablet_service.cc:1467] Processing CreateTablet for tablet ea66fe3d0da04d18b961927a9c988a20 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.478526 31429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0399a419a25c4981a2ee90d76cbf59b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.478673 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea66fe3d0da04d18b961927a9c988a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.478783 32263 tablet_bootstrap.cc:492] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:35.479528 32261 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:35.479672 32263 tablet_bootstrap.cc:654] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.480358 32261 tablet_bootstrap.cc:654] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.480521 31300 tablet_service.cc:1467] Processing CreateTablet for tablet 0399a419a25c4981a2ee90d76cbf59b1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.480718 31300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0399a419a25c4981a2ee90d76cbf59b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.480661 31562 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2c173b2c32494eb16e46a51ca36e31 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.480872 31561 tablet_service.cc:1467] Processing CreateTablet for tablet ea66fe3d0da04d18b961927a9c988a20 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.480944 31562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2c173b2c32494eb16e46a51ca36e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.481065 31561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea66fe3d0da04d18b961927a9c988a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.483144 32263 tablet_bootstrap.cc:492] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:35.483191 32261 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:35.483240 32263 ts_tablet_manager.cc:1367] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:35.483263 32261 ts_tablet_manager.cc:1367] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:35.483608 32261 raft_consensus.cc:348] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.483697 32261 raft_consensus.cc:374] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.483726 32261 raft_consensus.cc:729] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.483700 32263 raft_consensus.cc:348] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:35.483799 32263 raft_consensus.cc:374] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.483827 32263 raft_consensus.cc:729] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.483983 32263 consensus_queue.cc:260] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:35.484265 32263 ts_tablet_manager.cc:1398] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:35.484519 32266 tablet_bootstrap.cc:492] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:35.484587 32262 tablet_bootstrap.cc:654] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.485003 32261 consensus_queue.cc:260] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.485342 32266 tablet_bootstrap.cc:654] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.485810 32262 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:35.485888 32262 ts_tablet_manager.cc:1367] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:35.486227 32261 ts_tablet_manager.cc:1398] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:35.486224 32262 raft_consensus.cc:348] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.486318 32262 raft_consensus.cc:374] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.486351 32262 raft_consensus.cc:729] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.486385 32261 tablet_bootstrap.cc:492] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:35.486460 32262 consensus_queue.cc:260] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.486723 32262 ts_tablet_manager.cc:1398] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.486891 32262 tablet_bootstrap.cc:492] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:35.487174 32261 tablet_bootstrap.cc:654] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.487699 32262 tablet_bootstrap.cc:654] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.488521 32261 tablet_bootstrap.cc:492] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:35.488593 32261 ts_tablet_manager.cc:1367] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:35.488770 32266 tablet_bootstrap.cc:492] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:35.488868 32262 tablet_bootstrap.cc:492] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:35.488935 32262 ts_tablet_manager.cc:1367] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:35.489115 32261 raft_consensus.cc:348] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.489185 32261 raft_consensus.cc:374] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.489203 32261 raft_consensus.cc:729] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.489295 32261 consensus_queue.cc:260] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.489343 32262 raft_consensus.cc:348] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.489434 32262 raft_consensus.cc:374] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.489442 32266 ts_tablet_manager.cc:1367] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:35.489463 32262 raft_consensus.cc:729] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.489605 32262 consensus_queue.cc:260] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.489943 32261 ts_tablet_manager.cc:1398] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:35.490096 32261 tablet_bootstrap.cc:492] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:35.490406 32262 ts_tablet_manager.cc:1398] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.489889 32266 raft_consensus.cc:348] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.490571 32262 tablet_bootstrap.cc:492] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:35.490568 32266 raft_consensus.cc:374] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.490603 32266 raft_consensus.cc:729] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.490746 32266 consensus_queue.cc:260] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.490896 32261 tablet_bootstrap.cc:654] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.491007 32266 ts_tablet_manager.cc:1398] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:09:35.491180 32266 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:35.492976 32261 tablet_bootstrap.cc:492] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:35.493094 32261 ts_tablet_manager.cc:1367] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:35.493448 32261 raft_consensus.cc:348] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:35.493527 32262 tablet_bootstrap.cc:654] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.493613 32261 raft_consensus.cc:374] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.493656 32261 raft_consensus.cc:729] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.493765 32266 tablet_bootstrap.cc:654] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.493815 32261 consensus_queue.cc:260] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:35.494419 32261 ts_tablet_manager.cc:1398] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.494532 32262 tablet_bootstrap.cc:492] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:35.494601 32262 ts_tablet_manager.cc:1367] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:35.494756 32266 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:35.494809 32266 ts_tablet_manager.cc:1367] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:35.495002 32262 raft_consensus.cc:348] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:35.495096 32262 raft_consensus.cc:374] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.495127 32262 raft_consensus.cc:729] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.495121 32266 raft_consensus.cc:348] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.495213 32266 raft_consensus.cc:374] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.495239 32266 raft_consensus.cc:729] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.495224 32262 consensus_queue.cc:260] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:35.495420 32266 consensus_queue.cc:260] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.495486 32262 ts_tablet_manager.cc:1398] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.495748 32266 ts_tablet_manager.cc:1398] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.533962 32219 raft_consensus.cc:482] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.534087 32219 raft_consensus.cc:504] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:35.534543 32219 leader_election.cc:288] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:35.534965 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399a419a25c4981a2ee90d76cbf59b1" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:09:35.535104 31189 raft_consensus.cc:2454] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:35.535391 31253 leader_election.cc:302] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 30e0b8849e3841ca9ab0de01e719ee4a; no voters:
I20240501 14:09:35.535575 32219 raft_consensus.cc:2781] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.535645 32219 raft_consensus.cc:482] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.535689 32219 raft_consensus.cc:3037] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.536562 32219 raft_consensus.cc:504] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:35.536933 32219 leader_election.cc:288] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Requested vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:35.537250 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399a419a25c4981a2ee90d76cbf59b1" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:35.537358 31189 raft_consensus.cc:3037] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.538091 31189 raft_consensus.cc:2454] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:35.538363 31253 leader_election.cc:302] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 30e0b8849e3841ca9ab0de01e719ee4a; no voters:
I20240501 14:09:35.538530 32219 raft_consensus.cc:2781] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.538607 32219 raft_consensus.cc:686] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Becoming Leader. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Running, Role: LEADER
I20240501 14:09:35.538743 32219 consensus_queue.cc:237] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:35.539182 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399a419a25c4981a2ee90d76cbf59b1" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:35.539316 31451 raft_consensus.cc:2454] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:35.539311 31450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399a419a25c4981a2ee90d76cbf59b1" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:35.539469 31450 raft_consensus.cc:3037] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.540381 31450 raft_consensus.cc:2454] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:35.540520 31041 catalog_manager.cc:5484] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66). New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:35.554592 32219 raft_consensus.cc:482] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.554718 32219 raft_consensus.cc:504] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.555095 32219 leader_election.cc:288] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:35.555543 31450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea66fe3d0da04d18b961927a9c988a20" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:35.555543 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea66fe3d0da04d18b961927a9c988a20" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:35.555689 31450 raft_consensus.cc:2454] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:35.555689 31582 raft_consensus.cc:2454] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:35.555984 31254 leader_election.cc:302] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, 6a6fcc099d1340c383e5a429643d5b7c, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:35.556190 32219 raft_consensus.cc:2781] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.556258 32219 raft_consensus.cc:482] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.556289 32219 raft_consensus.cc:3037] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.557204 32219 raft_consensus.cc:504] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.557509 32219 leader_election.cc:288] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Requested vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:35.557832 31450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea66fe3d0da04d18b961927a9c988a20" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:35.557837 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea66fe3d0da04d18b961927a9c988a20" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:35.557953 31582 raft_consensus.cc:3037] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.557953 31450 raft_consensus.cc:3037] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.558840 31582 raft_consensus.cc:2454] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:35.558840 31450 raft_consensus.cc:2454] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:35.559154 31255 leader_election.cc:302] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:35.559429 32219 raft_consensus.cc:2781] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.559587 32219 raft_consensus.cc:686] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Becoming Leader. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Running, Role: LEADER
I20240501 14:09:35.559764 32219 consensus_queue.cc:237] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.561342 31041 catalog_manager.cc:5484] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66). New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:35.622834 32189 raft_consensus.cc:482] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.622974 32189 raft_consensus.cc:504] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.624027 32189 leader_election.cc:288] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:35.628459 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2c173b2c32494eb16e46a51ca36e31" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" is_pre_election: true
I20240501 14:09:35.628510 31450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2c173b2c32494eb16e46a51ca36e31" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:35.628626 31450 raft_consensus.cc:2454] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:35.628626 31320 raft_consensus.cc:2454] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:35.629420 31516 leader_election.cc:302] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a6fcc099d1340c383e5a429643d5b7c, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:35.629663 32189 raft_consensus.cc:2781] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.629724 32189 raft_consensus.cc:482] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.629770 32189 raft_consensus.cc:3037] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.630558 32189 raft_consensus.cc:504] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.630854 32189 leader_election.cc:288] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Requested vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:35.631168 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2c173b2c32494eb16e46a51ca36e31" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:35.631273 31320 raft_consensus.cc:3037] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.631222 31450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2c173b2c32494eb16e46a51ca36e31" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:35.631379 31450 raft_consensus.cc:3037] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.632030 31320 raft_consensus.cc:2454] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:35.632136 31450 raft_consensus.cc:2454] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:35.632361 31518 leader_election.cc:302] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:35.632529 32189 raft_consensus.cc:2781] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.632723 32189 raft_consensus.cc:686] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Becoming Leader. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Running, Role: LEADER
I20240501 14:09:35.633015 32189 consensus_queue.cc:237] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.635191 31041 catalog_manager.cc:5484] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: term changed from 0 to 1, leader changed from <none> to e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68). New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:35.783663 32281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:35.783900 32281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:35.786914 32281 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:09:35.876574 31450 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.876796 31582 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.877105 31189 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.877151 31451 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.877501 31450 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.877544 32219 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.877727 32193 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.877975 32193 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.878525 31320 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.878931 32189 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.884258 32189 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.885730 32193 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 123.803 ms
time per row: 0.0619013 ms
I20240501 14:09:35.948899 31044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52370:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:35.954262 31562 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8200d9c20e4140906c71c746badb94 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.954506 31300 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8200d9c20e4140906c71c746badb94 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.954603 31562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8200d9c20e4140906c71c746badb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.954792 31300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8200d9c20e4140906c71c746badb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.954785 31167 tablet_service.cc:1467] Processing CreateTablet for tablet ad0dac76fd0c4ae6b8e0549376c98e5d (DEFAULT_TABLE table=rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.955042 31167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad0dac76fd0c4ae6b8e0549376c98e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.954993 31169 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8200d9c20e4140906c71c746badb94 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.955211 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8200d9c20e4140906c71c746badb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.956760 32266 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:35.956804 32261 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:35.957726 32261 tablet_bootstrap.cc:654] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.957463 32263 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:35.957751 31429 tablet_service.cc:1467] Processing CreateTablet for tablet ad0dac76fd0c4ae6b8e0549376c98e5d (DEFAULT_TABLE table=rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.958019 31429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad0dac76fd0c4ae6b8e0549376c98e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.958098 32266 tablet_bootstrap.cc:654] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.958604 32263 tablet_bootstrap.cc:654] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.959216 32266 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:35.959295 32266 ts_tablet_manager.cc:1367] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:35.959746 32266 raft_consensus.cc:348] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:35.959849 32266 raft_consensus.cc:374] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.959882 32266 raft_consensus.cc:729] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.960024 32262 tablet_bootstrap.cc:492] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:35.960001 32266 consensus_queue.cc:260] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:35.960265 32266 ts_tablet_manager.cc:1398] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:35.960726 31167 tablet_service.cc:1467] Processing CreateTablet for tablet 8f91909d3b1b46e9b858ce5c5b2e7975 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.960911 31300 tablet_service.cc:1467] Processing CreateTablet for tablet 8f91909d3b1b46e9b858ce5c5b2e7975 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.960986 31167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f91909d3b1b46e9b858ce5c5b2e7975. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.961133 31300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f91909d3b1b46e9b858ce5c5b2e7975. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.961309 31562 tablet_service.cc:1467] Processing CreateTablet for tablet ad0dac76fd0c4ae6b8e0549376c98e5d (DEFAULT_TABLE table=rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.961519 31562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad0dac76fd0c4ae6b8e0549376c98e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.963215 31561 tablet_service.cc:1467] Processing CreateTablet for tablet 8f91909d3b1b46e9b858ce5c5b2e7975 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.963320 32266 tablet_bootstrap.cc:492] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:35.963435 31561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f91909d3b1b46e9b858ce5c5b2e7975. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.964183 32266 tablet_bootstrap.cc:654] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.965018 32261 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:35.965106 32261 ts_tablet_manager.cc:1367] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:35.965219 32266 tablet_bootstrap.cc:492] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:35.965287 32266 ts_tablet_manager.cc:1367] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:35.965289 32262 tablet_bootstrap.cc:654] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.965447 32261 raft_consensus.cc:348] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:35.965530 32261 raft_consensus.cc:374] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.965550 32261 raft_consensus.cc:729] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.965651 32261 consensus_queue.cc:260] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:35.965678 32266 raft_consensus.cc:348] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.965763 32266 raft_consensus.cc:374] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.965797 32266 raft_consensus.cc:729] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.965901 32261 ts_tablet_manager.cc:1398] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.965898 32266 consensus_queue.cc:260] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.966058 32261 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:35.966365 32262 tablet_bootstrap.cc:492] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:35.966444 32262 ts_tablet_manager.cc:1367] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:35.966789 32261 tablet_bootstrap.cc:654] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.966888 32262 raft_consensus.cc:348] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.966990 32262 raft_consensus.cc:374] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.967025 32262 raft_consensus.cc:729] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.967140 32262 consensus_queue.cc:260] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.967411 32262 ts_tablet_manager.cc:1398] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.967500 32263 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:35.967576 32263 ts_tablet_manager.cc:1367] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:35.967710 32266 ts_tablet_manager.cc:1398] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:35.967878 32266 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:35.967938 32263 raft_consensus.cc:348] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:35.968050 32263 raft_consensus.cc:374] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.968083 32263 raft_consensus.cc:729] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.968199 32263 consensus_queue.cc:260] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:35.968438 32263 ts_tablet_manager.cc:1398] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.968591 32263 tablet_bootstrap.cc:492] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:35.968927 32266 tablet_bootstrap.cc:654] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.969385 32263 tablet_bootstrap.cc:654] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.969998 32266 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:35.970115 32266 ts_tablet_manager.cc:1367] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:35.970614 32261 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:35.970569 32266 raft_consensus.cc:348] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.970706 32261 ts_tablet_manager.cc:1367] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:35.970774 32266 raft_consensus.cc:374] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.970858 32266 raft_consensus.cc:729] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.971042 32266 consensus_queue.cc:260] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.971135 32261 raft_consensus.cc:348] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.971253 32261 raft_consensus.cc:374] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.971297 32261 raft_consensus.cc:729] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.971411 32261 consensus_queue.cc:260] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.971539 32266 ts_tablet_manager.cc:1398] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:35.971665 32261 ts_tablet_manager.cc:1398] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.971771 32263 tablet_bootstrap.cc:492] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:35.971850 32263 ts_tablet_manager.cc:1367] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:35.972237 32263 raft_consensus.cc:348] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.972335 32263 raft_consensus.cc:374] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.972366 32263 raft_consensus.cc:729] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.972465 32263 consensus_queue.cc:260] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.972704 32263 ts_tablet_manager.cc:1398] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.972904 32263 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:35.973753 32263 tablet_bootstrap.cc:654] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.974637 32263 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:35.974709 32263 ts_tablet_manager.cc:1367] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:35.975028 32263 raft_consensus.cc:348] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.975109 32263 raft_consensus.cc:374] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.975138 32263 raft_consensus.cc:729] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.975265 32263 consensus_queue.cc:260] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:35.975525 32263 ts_tablet_manager.cc:1398] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.164620 32193 raft_consensus.cc:482] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.164762 32193 raft_consensus.cc:504] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:36.165251 32193 leader_election.cc:288] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:36.165695 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8200d9c20e4140906c71c746badb94" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:09:36.165699 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8200d9c20e4140906c71c746badb94" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:36.165833 31582 raft_consensus.cc:2454] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:36.165830 31189 raft_consensus.cc:2454] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:36.166146 31255 leader_election.cc:302] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:36.166312 32193 raft_consensus.cc:2781] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.166378 32193 raft_consensus.cc:482] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.166424 32193 raft_consensus.cc:3037] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.167196 32193 raft_consensus.cc:504] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:36.167492 32193 leader_election.cc:288] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Requested vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:36.167831 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8200d9c20e4140906c71c746badb94" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:36.167861 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8200d9c20e4140906c71c746badb94" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:36.167956 31582 raft_consensus.cc:3037] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.167956 31189 raft_consensus.cc:3037] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.168887 31189 raft_consensus.cc:2454] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:36.168887 31582 raft_consensus.cc:2454] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:36.169193 31253 leader_election.cc:302] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 30e0b8849e3841ca9ab0de01e719ee4a; no voters:
I20240501 14:09:36.169389 32193 raft_consensus.cc:2781] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.169459 32193 raft_consensus.cc:686] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Becoming Leader. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Running, Role: LEADER
I20240501 14:09:36.169593 32193 consensus_queue.cc:237] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:36.171149 31044 catalog_manager.cc:5484] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66). New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:36.280102 32184 raft_consensus.cc:482] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.280244 32184 raft_consensus.cc:504] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.280761 32184 leader_election.cc:288] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:36.281217 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0dac76fd0c4ae6b8e0549376c98e5d" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:09:36.281378 31189 raft_consensus.cc:2454] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 0.
I20240501 14:09:36.281697 31384 leader_election.cc:302] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:36.281889 32184 raft_consensus.cc:2781] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.281942 32184 raft_consensus.cc:482] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.281978 32184 raft_consensus.cc:3037] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.282742 32184 raft_consensus.cc:504] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.283068 32184 leader_election.cc:288] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 election: Requested vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:36.283413 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0dac76fd0c4ae6b8e0549376c98e5d" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:36.283538 31189 raft_consensus.cc:3037] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.284371 31189 raft_consensus.cc:2454] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 1.
I20240501 14:09:36.284585 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0dac76fd0c4ae6b8e0549376c98e5d" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:36.284672 31384 leader_election.cc:302] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:36.284653 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0dac76fd0c4ae6b8e0549376c98e5d" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:36.284725 31582 raft_consensus.cc:2454] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 0.
I20240501 14:09:36.284868 32184 raft_consensus.cc:2781] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.284955 32184 raft_consensus.cc:686] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Becoming Leader. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Running, Role: LEADER
I20240501 14:09:36.285085 32184 consensus_queue.cc:237] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.286741 31044 catalog_manager.cc:5484] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67). New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:36.309792 32193 raft_consensus.cc:482] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.309907 32193 raft_consensus.cc:504] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.310240 32193 leader_election.cc:288] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:36.310590 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f91909d3b1b46e9b858ce5c5b2e7975" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:09:36.310596 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f91909d3b1b46e9b858ce5c5b2e7975" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:36.310714 31582 raft_consensus.cc:2454] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:36.310722 31189 raft_consensus.cc:2454] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:36.310987 31253 leader_election.cc:302] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 30e0b8849e3841ca9ab0de01e719ee4a; no voters:
I20240501 14:09:36.311149 32193 raft_consensus.cc:2781] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.311208 32193 raft_consensus.cc:482] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.311246 32193 raft_consensus.cc:3037] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.311998 32193 raft_consensus.cc:504] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.312309 32193 leader_election.cc:288] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Requested vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:36.312585 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f91909d3b1b46e9b858ce5c5b2e7975" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:36.312655 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f91909d3b1b46e9b858ce5c5b2e7975" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:36.312700 31189 raft_consensus.cc:3037] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.312767 31582 raft_consensus.cc:3037] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.313628 31189 raft_consensus.cc:2454] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:36.313628 31582 raft_consensus.cc:2454] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:36.313895 31253 leader_election.cc:302] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 30e0b8849e3841ca9ab0de01e719ee4a; no voters:
I20240501 14:09:36.314074 32193 raft_consensus.cc:2781] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.314157 32193 raft_consensus.cc:686] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Becoming Leader. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Running, Role: LEADER
I20240501 14:09:36.314282 32193 consensus_queue.cc:237] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.315893 31044 catalog_manager.cc:5484] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66). New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:36.430740 32327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:36.430968 32327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:36.433936 32327 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:09:36.519098 31582 raft_consensus.cc:3037] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.519707 31189 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.520325 31582 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.520750 31582 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.521260 32193 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.521884 32200 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.522317 31581 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.522635 31188 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.522964 32200 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.523639 31189 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.523967 32193 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.524461 32219 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.524482 32193 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 116.396 ms
time per row: 0.058198 ms
I20240501 14:09:36.589324 31041 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52370:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:36.594794 31300 tablet_service.cc:1467] Processing CreateTablet for tablet c0b7ede9ffaf496e9b106b5d05ed25a5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=351ff5b1095247928fbb77893616e4a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.594904 31429 tablet_service.cc:1467] Processing CreateTablet for tablet c0b7ede9ffaf496e9b106b5d05ed25a5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=351ff5b1095247928fbb77893616e4a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.595115 31300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b7ede9ffaf496e9b106b5d05ed25a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.595206 31429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b7ede9ffaf496e9b106b5d05ed25a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.595834 31169 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffdb76b2c5149a18d45bee9581e605a (DEFAULT_TABLE table=rebalance_test_table_3 [id=351ff5b1095247928fbb77893616e4a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.596097 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffdb76b2c5149a18d45bee9581e605a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.597632 31167 tablet_service.cc:1467] Processing CreateTablet for tablet c0b7ede9ffaf496e9b106b5d05ed25a5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=351ff5b1095247928fbb77893616e4a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.597852 31167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b7ede9ffaf496e9b106b5d05ed25a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.598242 32392 tablet_bootstrap.cc:492] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:36.598771 31300 tablet_service.cc:1467] Processing CreateTablet for tablet 2e68be1521964ba8b864801fdf3b0df3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=351ff5b1095247928fbb77893616e4a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.599012 31300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e68be1521964ba8b864801fdf3b0df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.599159 31562 tablet_service.cc:1467] Processing CreateTablet for tablet 2e68be1521964ba8b864801fdf3b0df3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=351ff5b1095247928fbb77893616e4a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.599226 32390 tablet_bootstrap.cc:492] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:36.599431 31562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e68be1521964ba8b864801fdf3b0df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.599784 31561 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffdb76b2c5149a18d45bee9581e605a (DEFAULT_TABLE table=rebalance_test_table_3 [id=351ff5b1095247928fbb77893616e4a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.599988 31561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffdb76b2c5149a18d45bee9581e605a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.600193 32390 tablet_bootstrap.cc:654] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.601397 32392 tablet_bootstrap.cc:654] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.601500 32394 tablet_bootstrap.cc:492] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:36.601965 31431 tablet_service.cc:1467] Processing CreateTablet for tablet 2e68be1521964ba8b864801fdf3b0df3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=351ff5b1095247928fbb77893616e4a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.602182 31431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e68be1521964ba8b864801fdf3b0df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.602319 32394 tablet_bootstrap.cc:654] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.602437 32391 tablet_bootstrap.cc:492] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:36.603242 32391 tablet_bootstrap.cc:654] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.603499 32394 tablet_bootstrap.cc:492] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:36.603593 32394 ts_tablet_manager.cc:1367] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:36.604053 32394 raft_consensus.cc:348] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.604172 32394 raft_consensus.cc:374] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.604139 31429 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffdb76b2c5149a18d45bee9581e605a (DEFAULT_TABLE table=rebalance_test_table_3 [id=351ff5b1095247928fbb77893616e4a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.604209 32394 raft_consensus.cc:729] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.604360 31429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffdb76b2c5149a18d45bee9581e605a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.604370 32394 consensus_queue.cc:260] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.604732 32394 ts_tablet_manager.cc:1398] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.604763 32391 tablet_bootstrap.cc:492] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:36.604944 32394 tablet_bootstrap.cc:492] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:36.604944 32391 ts_tablet_manager.cc:1367] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:36.605454 32391 raft_consensus.cc:348] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.605616 32391 raft_consensus.cc:374] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.605697 32391 raft_consensus.cc:729] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.605753 32394 tablet_bootstrap.cc:654] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.605903 32391 consensus_queue.cc:260] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.606240 32391 ts_tablet_manager.cc:1398] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.606424 32391 tablet_bootstrap.cc:492] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:36.607120 32392 tablet_bootstrap.cc:492] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:36.607213 32392 ts_tablet_manager.cc:1367] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:36.607228 32391 tablet_bootstrap.cc:654] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.607322 32390 tablet_bootstrap.cc:492] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:36.607407 32390 ts_tablet_manager.cc:1367] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:36.607656 32392 raft_consensus.cc:348] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.607753 32392 raft_consensus.cc:374] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.607781 32392 raft_consensus.cc:729] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.607846 32390 raft_consensus.cc:348] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.607939 32390 raft_consensus.cc:374] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.607903 32392 consensus_queue.cc:260] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.607968 32390 raft_consensus.cc:729] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.608076 32390 consensus_queue.cc:260] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.608160 32392 ts_tablet_manager.cc:1398] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.608330 32392 tablet_bootstrap.cc:492] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:36.608327 32390 ts_tablet_manager.cc:1398] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:36.608475 32390 tablet_bootstrap.cc:492] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:36.609344 32390 tablet_bootstrap.cc:654] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.609441 32392 tablet_bootstrap.cc:654] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.610605 32392 tablet_bootstrap.cc:492] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:36.610678 32392 ts_tablet_manager.cc:1367] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:36.611070 32392 raft_consensus.cc:348] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.611164 32392 raft_consensus.cc:374] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.611193 32392 raft_consensus.cc:729] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.611331 32390 tablet_bootstrap.cc:492] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:36.611413 32390 ts_tablet_manager.cc:1367] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:36.611605 32391 tablet_bootstrap.cc:492] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:36.611680 32391 ts_tablet_manager.cc:1367] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:36.611737 32394 tablet_bootstrap.cc:492] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:36.611833 32394 ts_tablet_manager.cc:1367] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:36.611809 32390 raft_consensus.cc:348] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.611894 32390 raft_consensus.cc:374] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.611923 32390 raft_consensus.cc:729] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.612032 32390 consensus_queue.cc:260] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.612280 32390 ts_tablet_manager.cc:1398] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:36.612248 32394 raft_consensus.cc:348] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.612340 32394 raft_consensus.cc:374] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.612370 32394 raft_consensus.cc:729] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.612483 32394 consensus_queue.cc:260] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.612118 32391 raft_consensus.cc:348] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.613106 32394 ts_tablet_manager.cc:1398] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.613134 32391 raft_consensus.cc:374] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.613173 32391 raft_consensus.cc:729] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.613282 32391 consensus_queue.cc:260] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.613759 32392 consensus_queue.cc:260] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.613886 32391 ts_tablet_manager.cc:1398] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:36.614058 32391 tablet_bootstrap.cc:492] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:36.614068 32392 ts_tablet_manager.cc:1398] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:36.614902 32391 tablet_bootstrap.cc:654] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.616768 32391 tablet_bootstrap.cc:492] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:36.616878 32391 ts_tablet_manager.cc:1367] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:36.617311 32391 raft_consensus.cc:348] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.617408 32391 raft_consensus.cc:374] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.617439 32391 raft_consensus.cc:729] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.617522 32391 consensus_queue.cc:260] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.617744 32391 ts_tablet_manager.cc:1398] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:36.637709 32365 raft_consensus.cc:482] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.637838 32365 raft_consensus.cc:504] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.638271 32365 leader_election.cc:288] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:36.638667 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e68be1521964ba8b864801fdf3b0df3" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:36.638672 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e68be1521964ba8b864801fdf3b0df3" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" is_pre_election: true
I20240501 14:09:36.638806 31320 raft_consensus.cc:2454] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:36.638809 31451 raft_consensus.cc:2454] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:36.639133 31516 leader_election.cc:302] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a6fcc099d1340c383e5a429643d5b7c, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:36.639297 32365 raft_consensus.cc:2781] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.639366 32365 raft_consensus.cc:482] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.639402 32365 raft_consensus.cc:3037] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.640167 32365 raft_consensus.cc:504] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.640488 32365 leader_election.cc:288] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Requested vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:36.640866 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e68be1521964ba8b864801fdf3b0df3" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:36.640879 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e68be1521964ba8b864801fdf3b0df3" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:36.640997 31320 raft_consensus.cc:3037] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.640997 31451 raft_consensus.cc:3037] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.641908 31451 raft_consensus.cc:2454] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:36.641909 31320 raft_consensus.cc:2454] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:36.642186 31518 leader_election.cc:302] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:36.642431 32365 raft_consensus.cc:2781] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.642525 32365 raft_consensus.cc:686] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Becoming Leader. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Running, Role: LEADER
I20240501 14:09:36.642663 32365 consensus_queue.cc:237] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:36.644410 31041 catalog_manager.cc:5484] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: term changed from 0 to 1, leader changed from <none> to e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68). New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:36.645090 32200 raft_consensus.cc:482] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.645179 32200 raft_consensus.cc:504] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.645530 32200 leader_election.cc:288] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:36.645886 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffdb76b2c5149a18d45bee9581e605a" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:09:36.646036 31189 raft_consensus.cc:2454] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 0.
I20240501 14:09:36.646421 31384 leader_election.cc:302] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:36.646441 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffdb76b2c5149a18d45bee9581e605a" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:36.646553 31580 raft_consensus.cc:2454] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 0.
I20240501 14:09:36.646591 32200 raft_consensus.cc:2781] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.646639 32200 raft_consensus.cc:482] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.646673 32200 raft_consensus.cc:3037] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.647473 32200 raft_consensus.cc:504] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.647754 32200 leader_election.cc:288] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 election: Requested vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:36.648041 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffdb76b2c5149a18d45bee9581e605a" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:36.648041 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffdb76b2c5149a18d45bee9581e605a" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:36.648134 31189 raft_consensus.cc:3037] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.648134 31580 raft_consensus.cc:3037] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.648793 31189 raft_consensus.cc:2454] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 1.
I20240501 14:09:36.648880 31580 raft_consensus.cc:2454] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 1.
I20240501 14:09:36.649075 31384 leader_election.cc:302] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:36.649268 32200 raft_consensus.cc:2781] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.649344 32200 raft_consensus.cc:686] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Becoming Leader. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Running, Role: LEADER
I20240501 14:09:36.649470 32200 consensus_queue.cc:237] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.650938 31041 catalog_manager.cc:5484] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67). New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:36.865869 32370 raft_consensus.cc:482] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.866019 32370 raft_consensus.cc:504] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.866523 32370 leader_election.cc:288] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:36.866983 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b7ede9ffaf496e9b106b5d05ed25a5" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:36.867151 31451 raft_consensus.cc:2454] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:36.867497 31123 leader_election.cc:302] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:36.867669 32370 raft_consensus.cc:2781] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.867725 32370 raft_consensus.cc:482] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.867751 32370 raft_consensus.cc:3037] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.868587 32370 raft_consensus.cc:504] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.868934 32370 leader_election.cc:288] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Requested vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:36.869285 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b7ede9ffaf496e9b106b5d05ed25a5" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:36.869429 31451 raft_consensus.cc:3037] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.870092 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b7ede9ffaf496e9b106b5d05ed25a5" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" is_pre_election: true
I20240501 14:09:36.870209 31451 raft_consensus.cc:2454] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:36.870219 31320 raft_consensus.cc:2454] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:36.870195 31319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b7ede9ffaf496e9b106b5d05ed25a5" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:36.870335 31319 raft_consensus.cc:3037] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.870728 31123 leader_election.cc:302] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:36.870909 32370 raft_consensus.cc:2781] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.871002 32370 raft_consensus.cc:686] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Becoming Leader. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Running, Role: LEADER
I20240501 14:09:36.871153 31319 raft_consensus.cc:2454] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:36.871131 32370 consensus_queue.cc:237] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:36.872776 31041 catalog_manager.cc:5484] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:36.997727 32407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:36.997968 32407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:37.001329 32407 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 14:09:37.087270 31189 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.087697 32184 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.088145 31580 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.089099 32184 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.091197 31451 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.091703 31319 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.092129 32370 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.092450 32370 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.096659 31320 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.097185 31450 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.097252 32313 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.097630 32313 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 107.246 ms
time per row: 0.053623 ms
I20240501 14:09:37.152562 31045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52370:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:37.157883 31300 tablet_service.cc:1467] Processing CreateTablet for tablet 28013d7df97943e09c27c3bb1794ee80 (DEFAULT_TABLE table=rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.157946 31167 tablet_service.cc:1467] Processing CreateTablet for tablet 28013d7df97943e09c27c3bb1794ee80 (DEFAULT_TABLE table=rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.158232 31300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28013d7df97943e09c27c3bb1794ee80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.158246 31167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28013d7df97943e09c27c3bb1794ee80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.158285 31429 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa7bdac7bd1408cbf41d894d40aebd4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.158543 31429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa7bdac7bd1408cbf41d894d40aebd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.159562 31562 tablet_service.cc:1467] Processing CreateTablet for tablet 72ad9c714b2b4fff94063d2838835173 (DEFAULT_TABLE table=rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.159819 31562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ad9c714b2b4fff94063d2838835173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.159997 31561 tablet_service.cc:1467] Processing CreateTablet for tablet 28013d7df97943e09c27c3bb1794ee80 (DEFAULT_TABLE table=rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.160200 31561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28013d7df97943e09c27c3bb1794ee80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.161077 32475 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:37.161058 31431 tablet_service.cc:1467] Processing CreateTablet for tablet 72ad9c714b2b4fff94063d2838835173 (DEFAULT_TABLE table=rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.161342 31431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ad9c714b2b4fff94063d2838835173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.162055 32476 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:37.162014 31168 tablet_service.cc:1467] Processing CreateTablet for tablet 72ad9c714b2b4fff94063d2838835173 (DEFAULT_TABLE table=rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.162237 31168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ad9c714b2b4fff94063d2838835173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.162905 32476 tablet_bootstrap.cc:654] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.162902 31300 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa7bdac7bd1408cbf41d894d40aebd4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.163129 31300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa7bdac7bd1408cbf41d894d40aebd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.163331 31169 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa7bdac7bd1408cbf41d894d40aebd4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.163560 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa7bdac7bd1408cbf41d894d40aebd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.164644 32475 tablet_bootstrap.cc:654] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.165421 32474 tablet_bootstrap.cc:492] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:37.163553 32477 tablet_bootstrap.cc:492] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:37.165771 32475 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:37.165853 32475 ts_tablet_manager.cc:1367] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:37.166213 32475 raft_consensus.cc:348] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.166270 32474 tablet_bootstrap.cc:654] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.166306 32475 raft_consensus.cc:374] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.166334 32475 raft_consensus.cc:729] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.166375 32477 tablet_bootstrap.cc:654] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.166478 32475 consensus_queue.cc:260] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.166772 32475 ts_tablet_manager.cc:1398] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:37.166921 32475 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:37.167318 32477 tablet_bootstrap.cc:492] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:37.167392 32477 ts_tablet_manager.cc:1367] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:37.167435 32474 tablet_bootstrap.cc:492] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:37.167503 32474 ts_tablet_manager.cc:1367] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:37.167769 32475 tablet_bootstrap.cc:654] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.167809 32477 raft_consensus.cc:348] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.167850 32474 raft_consensus.cc:348] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.167902 32477 raft_consensus.cc:374] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.167930 32477 raft_consensus.cc:729] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.167930 32474 raft_consensus.cc:374] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.167973 32474 raft_consensus.cc:729] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.168049 32477 consensus_queue.cc:260] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.168097 32474 consensus_queue.cc:260] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.168306 32477 ts_tablet_manager.cc:1398] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.168468 32477 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:37.168715 32474 ts_tablet_manager.cc:1398] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.168941 32474 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:37.169256 32477 tablet_bootstrap.cc:654] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.169744 32474 tablet_bootstrap.cc:654] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.170359 32477 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:37.170498 32477 ts_tablet_manager.cc:1367] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:37.170944 32477 raft_consensus.cc:348] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.171108 32477 raft_consensus.cc:374] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.171191 32477 raft_consensus.cc:729] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.171376 32477 consensus_queue.cc:260] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.171453 32474 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:37.171463 32475 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:37.171523 32474 ts_tablet_manager.cc:1367] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:37.171527 32475 ts_tablet_manager.cc:1367] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:37.171926 32474 raft_consensus.cc:348] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.171926 32475 raft_consensus.cc:348] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.172024 32475 raft_consensus.cc:374] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.172024 32474 raft_consensus.cc:374] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.172053 32475 raft_consensus.cc:729] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.172053 32474 raft_consensus.cc:729] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.172163 32475 consensus_queue.cc:260] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.172181 32474 consensus_queue.cc:260] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.172420 32475 ts_tablet_manager.cc:1398] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.172793 32474 ts_tablet_manager.cc:1398] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.173570 32477 ts_tablet_manager.cc:1398] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:37.173871 32476 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:37.173986 32476 ts_tablet_manager.cc:1367] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:37.174156 32477 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:37.174542 32476 raft_consensus.cc:348] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.174721 32476 raft_consensus.cc:374] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.174801 32476 raft_consensus.cc:729] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.175009 32476 consensus_queue.cc:260] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.175107 32477 tablet_bootstrap.cc:654] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.175375 32476 ts_tablet_manager.cc:1398] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.175585 32476 tablet_bootstrap.cc:492] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:37.176465 32476 tablet_bootstrap.cc:654] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.176528 32477 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:37.176612 32477 ts_tablet_manager.cc:1367] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:37.177002 32477 raft_consensus.cc:348] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.177102 32477 raft_consensus.cc:374] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.177135 32477 raft_consensus.cc:729] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.177249 32477 consensus_queue.cc:260] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.177511 32477 ts_tablet_manager.cc:1398] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.177565 32476 tablet_bootstrap.cc:492] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:37.177639 32476 ts_tablet_manager.cc:1367] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:37.178045 32476 raft_consensus.cc:348] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.178145 32476 raft_consensus.cc:374] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.178184 32476 raft_consensus.cc:729] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.178316 32476 consensus_queue.cc:260] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.178648 32476 ts_tablet_manager.cc:1398] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.221489 32462 raft_consensus.cc:482] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.221616 32462 raft_consensus.cc:504] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.222054 32462 leader_election.cc:288] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:37.222486 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aa7bdac7bd1408cbf41d894d40aebd4" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" is_pre_election: true
I20240501 14:09:37.222486 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aa7bdac7bd1408cbf41d894d40aebd4" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:09:37.222637 31320 raft_consensus.cc:2454] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 0.
I20240501 14:09:37.222637 31189 raft_consensus.cc:2454] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 0.
I20240501 14:09:37.222970 31384 leader_election.cc:302] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 30e0b8849e3841ca9ab0de01e719ee4a, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:37.223153 32462 raft_consensus.cc:2781] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.223223 32462 raft_consensus.cc:482] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.223249 32462 raft_consensus.cc:3037] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.224071 32462 raft_consensus.cc:504] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.224388 32462 leader_election.cc:288] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 election: Requested vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:37.224699 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aa7bdac7bd1408cbf41d894d40aebd4" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:37.224784 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aa7bdac7bd1408cbf41d894d40aebd4" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:37.224856 31320 raft_consensus.cc:3037] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.224920 31189 raft_consensus.cc:3037] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.225781 31189 raft_consensus.cc:2454] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 1.
I20240501 14:09:37.225781 31320 raft_consensus.cc:2454] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 1.
I20240501 14:09:37.226059 31387 leader_election.cc:302] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:37.226230 32462 raft_consensus.cc:2781] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.226311 32462 raft_consensus.cc:686] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Becoming Leader. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Running, Role: LEADER
I20240501 14:09:37.226444 32462 consensus_queue.cc:237] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.228075 31046 catalog_manager.cc:5484] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67). New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:37.233687 32186 raft_consensus.cc:482] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.233791 32186 raft_consensus.cc:504] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.234148 32186 leader_election.cc:288] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:37.234508 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28013d7df97943e09c27c3bb1794ee80" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" is_pre_election: true
I20240501 14:09:37.234551 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28013d7df97943e09c27c3bb1794ee80" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:37.234633 31320 raft_consensus.cc:2454] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:37.234669 31580 raft_consensus.cc:2454] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:37.234910 31125 leader_election.cc:302] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 30e0b8849e3841ca9ab0de01e719ee4a; no voters:
I20240501 14:09:37.235064 32186 raft_consensus.cc:2781] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.235123 32186 raft_consensus.cc:482] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.235168 32186 raft_consensus.cc:3037] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.235918 32186 raft_consensus.cc:504] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.236234 32186 leader_election.cc:288] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Requested vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:37.236522 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28013d7df97943e09c27c3bb1794ee80" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:37.236554 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28013d7df97943e09c27c3bb1794ee80" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:37.236624 31320 raft_consensus.cc:3037] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.236649 31580 raft_consensus.cc:3037] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.237356 31580 raft_consensus.cc:2454] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:37.237360 31320 raft_consensus.cc:2454] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:37.237605 31125 leader_election.cc:302] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 30e0b8849e3841ca9ab0de01e719ee4a; no voters:
I20240501 14:09:37.237804 32186 raft_consensus.cc:2781] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.237879 32186 raft_consensus.cc:686] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Becoming Leader. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Running, Role: LEADER
I20240501 14:09:37.238006 32186 consensus_queue.cc:237] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.239617 31046 catalog_manager.cc:5484] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:37.333176 32365 raft_consensus.cc:482] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.333290 32365 raft_consensus.cc:504] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.333727 32365 leader_election.cc:288] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:37.334154 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ad9c714b2b4fff94063d2838835173" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:37.334313 31451 raft_consensus.cc:2454] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:37.334604 31516 leader_election.cc:302] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a6fcc099d1340c383e5a429643d5b7c, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:37.334785 32365 raft_consensus.cc:2781] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.334841 32365 raft_consensus.cc:482] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.334867 32365 raft_consensus.cc:3037] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.335845 32365 raft_consensus.cc:504] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.336192 32365 leader_election.cc:288] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Requested vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:37.336496 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ad9c714b2b4fff94063d2838835173" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:37.336601 31451 raft_consensus.cc:3037] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.337591 31451 raft_consensus.cc:2454] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:37.337818 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ad9c714b2b4fff94063d2838835173" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:37.337867 31516 leader_election.cc:302] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a6fcc099d1340c383e5a429643d5b7c, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:37.337932 31187 raft_consensus.cc:3037] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.337936 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ad9c714b2b4fff94063d2838835173" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:09:37.338245 32365 raft_consensus.cc:2781] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.338325 32365 raft_consensus.cc:686] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Becoming Leader. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Running, Role: LEADER
I20240501 14:09:37.338446 32365 consensus_queue.cc:237] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:37.338842 31187 raft_consensus.cc:2454] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:37.340097 31046 catalog_manager.cc:5484] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: term changed from 0 to 1, leader changed from <none> to e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68). New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:37.461656 32491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:37.461887 32491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:37.465027 32491 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 14:09:37.556499 31580 raft_consensus.cc:1261] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.557097 32370 consensus_queue.cc:1035] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.557893 31320 raft_consensus.cc:1261] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.558784 31320 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.559193 32462 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.559767 32370 consensus_queue.cc:1035] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.561151 31187 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.561980 32200 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.562088 31451 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.562491 32313 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.562814 31187 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.563196 32313 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.586277 32452 mvcc.cc:205] Tried to move back new op lower bound from 7022889277678112768 to 7022889276779560960. Current Snapshot: MvccSnapshot[applied={T|T < 7022889277678112768}]
INSERT report
rows total: 2000
time total: 119.682 ms
time per row: 0.0598408 ms
I20240501 14:09:37.624910 31046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52370:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:37.630141 31561 tablet_service.cc:1467] Processing CreateTablet for tablet f8cc4d3985224178aa008b4c9de5625b (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd036c60a12148d2ada73beb29aedcbf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.630201 31300 tablet_service.cc:1467] Processing CreateTablet for tablet f8cc4d3985224178aa008b4c9de5625b (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd036c60a12148d2ada73beb29aedcbf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.630467 31561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cc4d3985224178aa008b4c9de5625b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.630482 31300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cc4d3985224178aa008b4c9de5625b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.630923 31298 tablet_service.cc:1467] Processing CreateTablet for tablet a7810938b8fb406e9c6ac438f54afd03 (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd036c60a12148d2ada73beb29aedcbf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.631139 31298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7810938b8fb406e9c6ac438f54afd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.632241 32476 tablet_bootstrap.cc:492] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:37.632464 32474 tablet_bootstrap.cc:492] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:37.632434 31299 tablet_service.cc:1467] Processing CreateTablet for tablet 6bcd6a2f3c014d5ca0e32e16cdbc4edc (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd036c60a12148d2ada73beb29aedcbf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.632922 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bcd6a2f3c014d5ca0e32e16cdbc4edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.633152 32476 tablet_bootstrap.cc:654] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.633257 32474 tablet_bootstrap.cc:654] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.634186 32474 tablet_bootstrap.cc:492] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:37.634179 31562 tablet_service.cc:1467] Processing CreateTablet for tablet a7810938b8fb406e9c6ac438f54afd03 (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd036c60a12148d2ada73beb29aedcbf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.634263 32474 ts_tablet_manager.cc:1367] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:37.634382 31562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7810938b8fb406e9c6ac438f54afd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.634688 32474 raft_consensus.cc:348] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:37.634795 32474 raft_consensus.cc:374] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.634825 32474 raft_consensus.cc:729] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.634924 32474 consensus_queue.cc:260] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:37.635095 31561 tablet_service.cc:1467] Processing CreateTablet for tablet 6bcd6a2f3c014d5ca0e32e16cdbc4edc (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd036c60a12148d2ada73beb29aedcbf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.635179 32474 ts_tablet_manager.cc:1398] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:37.635219 31431 tablet_service.cc:1467] Processing CreateTablet for tablet f8cc4d3985224178aa008b4c9de5625b (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd036c60a12148d2ada73beb29aedcbf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.635298 31561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bcd6a2f3c014d5ca0e32e16cdbc4edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.635334 32474 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:37.635470 31431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cc4d3985224178aa008b4c9de5625b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.636098 32474 tablet_bootstrap.cc:654] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.637645 31429 tablet_service.cc:1467] Processing CreateTablet for tablet 6bcd6a2f3c014d5ca0e32e16cdbc4edc (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd036c60a12148d2ada73beb29aedcbf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.637843 31429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bcd6a2f3c014d5ca0e32e16cdbc4edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.638062 32474 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:37.638131 32474 ts_tablet_manager.cc:1367] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:37.638537 32474 raft_consensus.cc:348] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.638617 32474 raft_consensus.cc:374] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.638638 32474 raft_consensus.cc:729] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.638733 32474 consensus_queue.cc:260] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.638970 32474 ts_tablet_manager.cc:1398] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:37.639107 32474 tablet_bootstrap.cc:492] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:37.639276 32475 tablet_bootstrap.cc:492] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:37.639825 32474 tablet_bootstrap.cc:654] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.640370 32475 tablet_bootstrap.cc:654] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.641084 32474 tablet_bootstrap.cc:492] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:37.641163 32474 ts_tablet_manager.cc:1367] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:37.641501 32474 raft_consensus.cc:348] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:37.641616 32474 raft_consensus.cc:374] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.641666 32474 raft_consensus.cc:729] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.641789 32474 consensus_queue.cc:260] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:37.642037 32474 ts_tablet_manager.cc:1398] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.642094 31169 tablet_service.cc:1467] Processing CreateTablet for tablet a7810938b8fb406e9c6ac438f54afd03 (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd036c60a12148d2ada73beb29aedcbf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.642364 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7810938b8fb406e9c6ac438f54afd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.644063 32477 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:37.644884 32475 tablet_bootstrap.cc:492] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:37.644923 32477 tablet_bootstrap.cc:654] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.644974 32475 ts_tablet_manager.cc:1367] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:37.645411 32475 raft_consensus.cc:348] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:37.645500 32475 raft_consensus.cc:374] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.645521 32475 raft_consensus.cc:729] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.645645 32475 consensus_queue.cc:260] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:37.645924 32475 ts_tablet_manager.cc:1398] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:37.645951 32477 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:37.646024 32477 ts_tablet_manager.cc:1367] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:37.646310 32475 tablet_bootstrap.cc:492] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:37.646416 32477 raft_consensus.cc:348] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.646526 32477 raft_consensus.cc:374] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.646565 32477 raft_consensus.cc:729] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.646687 32477 consensus_queue.cc:260] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.646942 32477 ts_tablet_manager.cc:1398] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:37.647168 32475 tablet_bootstrap.cc:654] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.648558 32475 tablet_bootstrap.cc:492] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:37.648644 32475 ts_tablet_manager.cc:1367] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:37.649052 32475 raft_consensus.cc:348] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:37.649137 32475 raft_consensus.cc:374] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.649158 32475 raft_consensus.cc:729] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.649255 32475 consensus_queue.cc:260] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:37.649515 32475 ts_tablet_manager.cc:1398] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.650182 32476 tablet_bootstrap.cc:492] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:37.650265 32476 ts_tablet_manager.cc:1367] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.000s
I20240501 14:09:37.650697 32476 raft_consensus.cc:348] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:37.650818 32476 raft_consensus.cc:374] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.650861 32476 raft_consensus.cc:729] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.650998 32476 consensus_queue.cc:260] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:37.651291 32476 ts_tablet_manager.cc:1398] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.651525 32476 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:37.652359 32476 tablet_bootstrap.cc:654] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.653389 32476 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:37.653477 32476 ts_tablet_manager.cc:1367] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:37.653882 32476 raft_consensus.cc:348] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.653992 32476 raft_consensus.cc:374] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.654034 32476 raft_consensus.cc:729] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.654161 32476 consensus_queue.cc:260] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.654417 32476 ts_tablet_manager.cc:1398] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.654596 32476 tablet_bootstrap.cc:492] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:37.655452 32476 tablet_bootstrap.cc:654] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.656455 32476 tablet_bootstrap.cc:492] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:37.656538 32476 ts_tablet_manager.cc:1367] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:37.656999 32476 raft_consensus.cc:348] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:37.657136 32476 raft_consensus.cc:374] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.657174 32476 raft_consensus.cc:729] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.657289 32476 consensus_queue.cc:260] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:37.657526 32476 ts_tablet_manager.cc:1398] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.811735 32430 raft_consensus.cc:482] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.811878 32430 raft_consensus.cc:504] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:37.812270 32430 leader_election.cc:288] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:37.812692 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cc4d3985224178aa008b4c9de5625b" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:37.812700 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cc4d3985224178aa008b4c9de5625b" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:37.812825 31580 raft_consensus.cc:2454] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:37.812865 31451 raft_consensus.cc:2454] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:37.813153 31255 leader_election.cc:302] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:37.813335 32430 raft_consensus.cc:2781] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.813405 32430 raft_consensus.cc:482] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.813455 32430 raft_consensus.cc:3037] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.814249 32430 raft_consensus.cc:504] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:37.814568 32430 leader_election.cc:288] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Requested vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:37.814894 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cc4d3985224178aa008b4c9de5625b" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:37.814935 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cc4d3985224178aa008b4c9de5625b" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:37.814996 31580 raft_consensus.cc:3037] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.815028 31451 raft_consensus.cc:3037] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.815721 31451 raft_consensus.cc:2454] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:37.815726 31580 raft_consensus.cc:2454] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:37.816000 31255 leader_election.cc:302] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:37.816159 32430 raft_consensus.cc:2781] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.816304 32430 raft_consensus.cc:686] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Becoming Leader. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Running, Role: LEADER
I20240501 14:09:37.816434 32430 consensus_queue.cc:237] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:37.818027 31044 catalog_manager.cc:5484] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66). New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:37.879441 32526 raft_consensus.cc:482] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.879577 32526 raft_consensus.cc:504] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.879983 32526 leader_election.cc:288] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:37.880352 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7810938b8fb406e9c6ac438f54afd03" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" is_pre_election: true
I20240501 14:09:37.880385 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7810938b8fb406e9c6ac438f54afd03" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:37.880489 31320 raft_consensus.cc:2454] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:37.880525 31580 raft_consensus.cc:2454] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:37.880790 31125 leader_election.cc:302] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 30e0b8849e3841ca9ab0de01e719ee4a; no voters:
I20240501 14:09:37.880971 32526 raft_consensus.cc:2781] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.881028 32526 raft_consensus.cc:482] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.881063 32526 raft_consensus.cc:3037] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.881824 32526 raft_consensus.cc:504] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.882138 32526 leader_election.cc:288] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Requested vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:37.882436 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7810938b8fb406e9c6ac438f54afd03" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:37.882459 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7810938b8fb406e9c6ac438f54afd03" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:37.882536 31320 raft_consensus.cc:3037] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.882551 31580 raft_consensus.cc:3037] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.883241 31580 raft_consensus.cc:2454] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:37.883493 31124 leader_election.cc:302] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:37.883641 32526 raft_consensus.cc:2781] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.883729 32526 raft_consensus.cc:686] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Becoming Leader. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Running, Role: LEADER
I20240501 14:09:37.883857 32526 consensus_queue.cc:237] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:37.885432 31044 catalog_manager.cc:5484] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:37.886714 32365 raft_consensus.cc:482] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.886811 32365 raft_consensus.cc:504] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:37.887166 32365 leader_election.cc:288] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:37.887506 31319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bcd6a2f3c014d5ca0e32e16cdbc4edc" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" is_pre_election: true
I20240501 14:09:37.887563 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bcd6a2f3c014d5ca0e32e16cdbc4edc" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:37.887634 31319 raft_consensus.cc:2454] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:37.887694 31451 raft_consensus.cc:2454] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:37.887892 31518 leader_election.cc:302] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:37.888110 32365 raft_consensus.cc:2781] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.888152 32365 raft_consensus.cc:482] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.888186 32365 raft_consensus.cc:3037] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.888818 32365 raft_consensus.cc:504] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:37.889149 32365 leader_election.cc:288] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Requested vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:37.889436 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bcd6a2f3c014d5ca0e32e16cdbc4edc" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:37.889479 31319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bcd6a2f3c014d5ca0e32e16cdbc4edc" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:37.889556 31319 raft_consensus.cc:3037] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.889561 31451 raft_consensus.cc:3037] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.890200 31451 raft_consensus.cc:2454] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:37.890473 31516 leader_election.cc:302] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a6fcc099d1340c383e5a429643d5b7c, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:37.890626 32365 raft_consensus.cc:2781] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.890710 32365 raft_consensus.cc:686] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Becoming Leader. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Running, Role: LEADER
I20240501 14:09:37.890831 32365 consensus_queue.cc:237] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:37.892241 31044 catalog_manager.cc:5484] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: term changed from 0 to 1, leader changed from <none> to e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68). New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:37.901525 31319 raft_consensus.cc:2454] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:37.901595 31320 raft_consensus.cc:2454] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
W20240501 14:09:38.045452 32540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:38.045702 32540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:38.049095 32540 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 14:09:38.124131 31580 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.124117 31320 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.124547 32186 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.124708 32526 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.126263 31451 raft_consensus.cc:1261] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.126612 31319 raft_consensus.cc:1261] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.127923 32313 consensus_queue.cc:1035] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.128558 32313 consensus_queue.cc:1035] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.131520 31581 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.131971 31450 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.133565 32430 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.135058 32575 mvcc.cc:205] Tried to move back new op lower bound from 7022889279991484416 to 7022889279013396480. Current Snapshot: MvccSnapshot[applied={T|T < 7022889279991484416}]
I20240501 14:09:38.135565 32577 mvcc.cc:205] Tried to move back new op lower bound from 7022889279991484416 to 7022889279013396480. Current Snapshot: MvccSnapshot[applied={T|T < 7022889279991484416}]
I20240501 14:09:38.138150 32430 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.141414 32576 mvcc.cc:205] Tried to move back new op lower bound from 7022889279991484416 to 7022889279013396480. Current Snapshot: MvccSnapshot[applied={T|T < 7022889279991484416 or (T in {7022889279991484416})}]
INSERT report
rows total: 2000
time total: 103.647 ms
time per row: 0.0518237 ms
I20240501 14:09:38.193372 31044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52370:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:38.199319 31168 tablet_service.cc:1467] Processing CreateTablet for tablet 96f5e4e25a674a99a6762757f0ab1545 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1f2efe25014e49d8b36ca467c0f40ef3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.199637 31168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f5e4e25a674a99a6762757f0ab1545. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.199597 31431 tablet_service.cc:1467] Processing CreateTablet for tablet 96f5e4e25a674a99a6762757f0ab1545 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1f2efe25014e49d8b36ca467c0f40ef3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.199872 31431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f5e4e25a674a99a6762757f0ab1545. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.199880 31169 tablet_service.cc:1467] Processing CreateTablet for tablet bca09920a7904362bc546916e3f6f5ac (DEFAULT_TABLE table=rebalance_test_table_6 [id=1f2efe25014e49d8b36ca467c0f40ef3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.200104 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca09920a7904362bc546916e3f6f5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.201685 32604 tablet_bootstrap.cc:492] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:38.202456 31429 tablet_service.cc:1467] Processing CreateTablet for tablet bca09920a7904362bc546916e3f6f5ac (DEFAULT_TABLE table=rebalance_test_table_6 [id=1f2efe25014e49d8b36ca467c0f40ef3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.202581 32604 tablet_bootstrap.cc:654] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.202700 31429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca09920a7904362bc546916e3f6f5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.203532 31299 tablet_service.cc:1467] Processing CreateTablet for tablet bca09920a7904362bc546916e3f6f5ac (DEFAULT_TABLE table=rebalance_test_table_6 [id=1f2efe25014e49d8b36ca467c0f40ef3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.203598 31298 tablet_service.cc:1467] Processing CreateTablet for tablet b67bd0eb45a3486492b75ca3ad3be524 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1f2efe25014e49d8b36ca467c0f40ef3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.203791 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca09920a7904362bc546916e3f6f5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.203833 31430 tablet_service.cc:1467] Processing CreateTablet for tablet b67bd0eb45a3486492b75ca3ad3be524 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1f2efe25014e49d8b36ca467c0f40ef3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.203899 31298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b67bd0eb45a3486492b75ca3ad3be524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.204046 31430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b67bd0eb45a3486492b75ca3ad3be524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.204787 31561 tablet_service.cc:1467] Processing CreateTablet for tablet 96f5e4e25a674a99a6762757f0ab1545 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1f2efe25014e49d8b36ca467c0f40ef3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.205178 31561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f5e4e25a674a99a6762757f0ab1545. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.206473 32604 tablet_bootstrap.cc:492] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:38.206568 32604 ts_tablet_manager.cc:1367] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:38.207031 32604 raft_consensus.cc:348] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.207170 32604 raft_consensus.cc:374] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.207221 32604 raft_consensus.cc:729] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.207270 32607 tablet_bootstrap.cc:492] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:38.207379 32604 consensus_queue.cc:260] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.208164 32604 ts_tablet_manager.cc:1398] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.208197 32607 tablet_bootstrap.cc:654] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.208354 32604 tablet_bootstrap.cc:492] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:38.209254 32604 tablet_bootstrap.cc:654] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.210358 32604 tablet_bootstrap.cc:492] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:38.210496 32604 ts_tablet_manager.cc:1367] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:38.210633 31169 tablet_service.cc:1467] Processing CreateTablet for tablet b67bd0eb45a3486492b75ca3ad3be524 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1f2efe25014e49d8b36ca467c0f40ef3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.210860 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b67bd0eb45a3486492b75ca3ad3be524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.210909 32604 raft_consensus.cc:348] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.210994 32604 raft_consensus.cc:374] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.211021 32604 raft_consensus.cc:729] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.211133 32604 consensus_queue.cc:260] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.211385 32604 ts_tablet_manager.cc:1398] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:38.212488 32604 tablet_bootstrap.cc:492] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:38.212981 32606 tablet_bootstrap.cc:492] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:38.213357 32604 tablet_bootstrap.cc:654] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.213835 32606 tablet_bootstrap.cc:654] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.214762 32604 tablet_bootstrap.cc:492] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:38.214841 32604 ts_tablet_manager.cc:1367] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.215229 32612 tablet_bootstrap.cc:492] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:38.215242 32604 raft_consensus.cc:348] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:38.215330 32604 raft_consensus.cc:374] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.215363 32604 raft_consensus.cc:729] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.215490 32604 consensus_queue.cc:260] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:38.215728 32604 ts_tablet_manager.cc:1398] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.216270 32606 tablet_bootstrap.cc:492] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:38.216409 32606 ts_tablet_manager.cc:1367] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:38.216888 32606 raft_consensus.cc:348] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:38.217063 32606 raft_consensus.cc:374] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.217140 32606 raft_consensus.cc:729] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.217185 32612 tablet_bootstrap.cc:654] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.218312 32612 tablet_bootstrap.cc:492] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:38.218392 32612 ts_tablet_manager.cc:1367] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:38.218577 32607 tablet_bootstrap.cc:492] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:38.218680 32607 ts_tablet_manager.cc:1367] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:38.218724 32606 consensus_queue.cc:260] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:38.218786 32612 raft_consensus.cc:348] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.218868 32612 raft_consensus.cc:374] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.218894 32612 raft_consensus.cc:729] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.219007 32612 consensus_queue.cc:260] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.219075 32606 ts_tablet_manager.cc:1398] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:38.219125 32607 raft_consensus.cc:348] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.219225 32607 raft_consensus.cc:374] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.219246 32612 ts_tablet_manager.cc:1398] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.219283 32606 tablet_bootstrap.cc:492] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:38.219283 32607 raft_consensus.cc:729] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.219378 32612 tablet_bootstrap.cc:492] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:38.219405 32607 consensus_queue.cc:260] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.219674 32607 ts_tablet_manager.cc:1398] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.220243 32612 tablet_bootstrap.cc:654] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.220965 32606 tablet_bootstrap.cc:654] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.222762 32612 tablet_bootstrap.cc:492] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:38.222868 32612 ts_tablet_manager.cc:1367] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:38.222965 32606 tablet_bootstrap.cc:492] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:38.223047 32606 ts_tablet_manager.cc:1367] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:38.223289 32612 raft_consensus.cc:348] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:38.223397 32612 raft_consensus.cc:374] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.223448 32612 raft_consensus.cc:729] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.223440 32606 raft_consensus.cc:348] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.223552 32606 raft_consensus.cc:374] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.223575 32612 consensus_queue.cc:260] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:38.223627 32606 raft_consensus.cc:729] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.223742 32606 consensus_queue.cc:260] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.224043 32606 ts_tablet_manager.cc:1398] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.223930 32612 ts_tablet_manager.cc:1398] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.225865 32612 tablet_bootstrap.cc:492] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:38.226743 32612 tablet_bootstrap.cc:654] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.227684 32612 tablet_bootstrap.cc:492] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:38.227757 32612 ts_tablet_manager.cc:1367] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.228088 32612 raft_consensus.cc:348] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.228173 32612 raft_consensus.cc:374] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.228201 32612 raft_consensus.cc:729] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.228314 32612 consensus_queue.cc:260] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.228546 32612 ts_tablet_manager.cc:1398] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:38.338245 32536 raft_consensus.cc:482] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.338382 32536 raft_consensus.cc:504] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:38.338810 32536 leader_election.cc:288] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:38.339226 31450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67bd0eb45a3486492b75ca3ad3be524" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:38.339273 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67bd0eb45a3486492b75ca3ad3be524" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:09:38.339360 31450 raft_consensus.cc:2454] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:38.339437 31187 raft_consensus.cc:2454] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:38.339658 31254 leader_election.cc:302] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:38.339828 32536 raft_consensus.cc:2781] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.339886 32536 raft_consensus.cc:482] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.339910 32536 raft_consensus.cc:3037] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.340730 32536 raft_consensus.cc:504] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:38.341126 32536 leader_election.cc:288] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Requested vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:38.341439 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67bd0eb45a3486492b75ca3ad3be524" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:38.341451 31450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67bd0eb45a3486492b75ca3ad3be524" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:38.341540 31187 raft_consensus.cc:3037] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.341562 31450 raft_consensus.cc:3037] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.342298 31187 raft_consensus.cc:2454] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:38.342361 31450 raft_consensus.cc:2454] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:38.342578 31253 leader_election.cc:302] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 30e0b8849e3841ca9ab0de01e719ee4a; no voters:
I20240501 14:09:38.342756 32536 raft_consensus.cc:2781] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.342851 32536 raft_consensus.cc:686] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Becoming Leader. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Running, Role: LEADER
I20240501 14:09:38.342991 32536 consensus_queue.cc:237] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:38.344713 31046 catalog_manager.cc:5484] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66). New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:38.387799 32186 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.387949 32186 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.388391 32186 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:38.388821 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:38.388850 31450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:38.388998 31580 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:38.389014 31450 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:38.389318 31124 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:38.389547 32186 raft_consensus.cc:2781] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.389612 32186 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.389664 32186 raft_consensus.cc:3037] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.390465 32186 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.390758 32186 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Requested vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:38.391032 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:38.391060 31450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:38.391127 31580 raft_consensus.cc:3037] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.391147 31450 raft_consensus.cc:3037] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.391830 31580 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:38.391866 31450 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:38.392109 31124 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:38.392262 32186 raft_consensus.cc:2781] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.392347 32186 raft_consensus.cc:686] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Becoming Leader. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Running, Role: LEADER
I20240501 14:09:38.392493 32186 consensus_queue.cc:237] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.394212 31046 catalog_manager.cc:5484] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.489626 32186 raft_consensus.cc:482] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.489739 32186 raft_consensus.cc:504] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.490094 32186 leader_election.cc:288] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:38.490491 31450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca09920a7904362bc546916e3f6f5ac" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:38.490494 31319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca09920a7904362bc546916e3f6f5ac" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" is_pre_election: true
I20240501 14:09:38.490617 31450 raft_consensus.cc:2454] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:38.490630 31319 raft_consensus.cc:2454] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:38.490916 31123 leader_election.cc:302] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:38.491086 32186 raft_consensus.cc:2781] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.491145 32186 raft_consensus.cc:482] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.491178 32186 raft_consensus.cc:3037] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.491943 32186 raft_consensus.cc:504] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.492256 32186 leader_election.cc:288] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Requested vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:38.492542 31319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca09920a7904362bc546916e3f6f5ac" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:38.492583 31450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca09920a7904362bc546916e3f6f5ac" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:38.492651 31319 raft_consensus.cc:3037] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.492710 31450 raft_consensus.cc:3037] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.493536 31319 raft_consensus.cc:2454] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:38.493543 31450 raft_consensus.cc:2454] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:38.493781 31123 leader_election.cc:302] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:38.493952 32186 raft_consensus.cc:2781] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.494035 32186 raft_consensus.cc:686] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Becoming Leader. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Running, Role: LEADER
I20240501 14:09:38.494164 32186 consensus_queue.cc:237] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.495878 31046 catalog_manager.cc:5484] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:38.606343 32620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:38.606617 32620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:38.609732 32620 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 14:09:38.703299 31319 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.703634 31450 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.704079 32186 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.704597 31187 raft_consensus.cc:1261] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.705053 32536 consensus_queue.cc:1035] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.705372 32186 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.705786 31450 raft_consensus.cc:1261] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.706144 32536 consensus_queue.cc:1035] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.707911 31580 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.710024 32526 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.713539 31449 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.715060 32186 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.295 ms
time per row: 0.0561474 ms
I20240501 14:09:38.767225 31045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52370:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:38.772395 31299 tablet_service.cc:1467] Processing CreateTablet for tablet 71241540f585474a8f62da16ed2d9874 (DEFAULT_TABLE table=rebalance_test_table_7 [id=855c959aa35441b989d386caf8f8eded]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.772435 31169 tablet_service.cc:1467] Processing CreateTablet for tablet 71241540f585474a8f62da16ed2d9874 (DEFAULT_TABLE table=rebalance_test_table_7 [id=855c959aa35441b989d386caf8f8eded]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.772730 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71241540f585474a8f62da16ed2d9874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.772751 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71241540f585474a8f62da16ed2d9874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.773352 31562 tablet_service.cc:1467] Processing CreateTablet for tablet 043344ce36a74a18a0d67ff5bd473e68 (DEFAULT_TABLE table=rebalance_test_table_7 [id=855c959aa35441b989d386caf8f8eded]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.773613 31562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043344ce36a74a18a0d67ff5bd473e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.774781 32681 tablet_bootstrap.cc:492] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:38.774888 31429 tablet_service.cc:1467] Processing CreateTablet for tablet 043344ce36a74a18a0d67ff5bd473e68 (DEFAULT_TABLE table=rebalance_test_table_7 [id=855c959aa35441b989d386caf8f8eded]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.775146 31429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043344ce36a74a18a0d67ff5bd473e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.775177 31561 tablet_service.cc:1467] Processing CreateTablet for tablet 71241540f585474a8f62da16ed2d9874 (DEFAULT_TABLE table=rebalance_test_table_7 [id=855c959aa35441b989d386caf8f8eded]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.775391 31561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71241540f585474a8f62da16ed2d9874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.775765 32681 tablet_bootstrap.cc:654] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.775816 31560 tablet_service.cc:1467] Processing CreateTablet for tablet c082ccc777ba4f9a937931f707645e54 (DEFAULT_TABLE table=rebalance_test_table_7 [id=855c959aa35441b989d386caf8f8eded]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.776024 31560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c082ccc777ba4f9a937931f707645e54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.776883 32681 tablet_bootstrap.cc:492] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:38.776962 32681 ts_tablet_manager.cc:1367] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:38.777093 31430 tablet_service.cc:1467] Processing CreateTablet for tablet c082ccc777ba4f9a937931f707645e54 (DEFAULT_TABLE table=rebalance_test_table_7 [id=855c959aa35441b989d386caf8f8eded]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.777307 31430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c082ccc777ba4f9a937931f707645e54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.777415 32681 raft_consensus.cc:348] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:38.777527 32681 raft_consensus.cc:374] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.777558 32681 raft_consensus.cc:729] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.777680 32681 consensus_queue.cc:260] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:38.777915 32681 ts_tablet_manager.cc:1398] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.778872 31299 tablet_service.cc:1467] Processing CreateTablet for tablet c082ccc777ba4f9a937931f707645e54 (DEFAULT_TABLE table=rebalance_test_table_7 [id=855c959aa35441b989d386caf8f8eded]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.779069 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c082ccc777ba4f9a937931f707645e54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.779273 32685 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:38.780156 31169 tablet_service.cc:1467] Processing CreateTablet for tablet 043344ce36a74a18a0d67ff5bd473e68 (DEFAULT_TABLE table=rebalance_test_table_7 [id=855c959aa35441b989d386caf8f8eded]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.780404 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043344ce36a74a18a0d67ff5bd473e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.780428 32685 tablet_bootstrap.cc:654] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.780963 32681 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:38.781289 32682 tablet_bootstrap.cc:492] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:38.781553 32685 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:38.781635 32685 ts_tablet_manager.cc:1367] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:38.781808 32681 tablet_bootstrap.cc:654] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.782061 32685 raft_consensus.cc:348] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.782162 32685 raft_consensus.cc:374] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.782193 32685 raft_consensus.cc:729] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.782199 32682 tablet_bootstrap.cc:654] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.782311 32685 consensus_queue.cc:260] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.782593 32685 ts_tablet_manager.cc:1398] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.783344 32682 tablet_bootstrap.cc:492] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:38.783442 32682 ts_tablet_manager.cc:1367] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:38.783895 32682 raft_consensus.cc:348] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:38.783996 32682 raft_consensus.cc:374] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.784029 32682 raft_consensus.cc:729] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.784186 32685 tablet_bootstrap.cc:492] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:38.784145 32682 consensus_queue.cc:260] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:38.784405 32682 ts_tablet_manager.cc:1398] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.784582 32682 tablet_bootstrap.cc:492] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:38.785126 32685 tablet_bootstrap.cc:654] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.785310 32683 tablet_bootstrap.cc:492] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:38.785401 32682 tablet_bootstrap.cc:654] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.786108 32683 tablet_bootstrap.cc:654] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.786473 32685 tablet_bootstrap.cc:492] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:38.786543 32685 ts_tablet_manager.cc:1367] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.786962 32685 raft_consensus.cc:348] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.787061 32685 raft_consensus.cc:374] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.787094 32685 raft_consensus.cc:729] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.787199 32685 consensus_queue.cc:260] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.787453 32685 ts_tablet_manager.cc:1398] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.787631 32681 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:38.787706 32681 ts_tablet_manager.cc:1367] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:38.788300 32683 tablet_bootstrap.cc:492] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:38.788295 32681 raft_consensus.cc:348] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.788379 32683 ts_tablet_manager.cc:1367] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:38.788393 32681 raft_consensus.cc:374] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.788424 32681 raft_consensus.cc:729] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.788568 32681 consensus_queue.cc:260] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.788779 32683 raft_consensus.cc:348] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.788918 32683 raft_consensus.cc:374] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.788955 32683 raft_consensus.cc:729] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.789489 32683 consensus_queue.cc:260] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.789578 32681 ts_tablet_manager.cc:1398] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.789734 32682 tablet_bootstrap.cc:492] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:38.789816 32682 ts_tablet_manager.cc:1367] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:38.789954 32683 ts_tablet_manager.cc:1398] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.790167 32683 tablet_bootstrap.cc:492] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:38.790234 32682 raft_consensus.cc:348] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.790330 32682 raft_consensus.cc:374] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.790362 32682 raft_consensus.cc:729] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.790796 32682 consensus_queue.cc:260] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.791258 32683 tablet_bootstrap.cc:654] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.791813 32682 ts_tablet_manager.cc:1398] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.792353 32683 tablet_bootstrap.cc:492] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:38.792443 32683 ts_tablet_manager.cc:1367] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:38.792867 32683 raft_consensus.cc:348] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:38.792961 32683 raft_consensus.cc:374] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.792994 32683 raft_consensus.cc:729] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.793109 32683 consensus_queue.cc:260] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:38.793363 32683 ts_tablet_manager.cc:1398] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:38.793653 32683 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:38.794437 32683 tablet_bootstrap.cc:654] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.795475 32683 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:38.795600 32683 ts_tablet_manager.cc:1367] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:38.796262 32683 raft_consensus.cc:348] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.796345 32683 raft_consensus.cc:374] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.796366 32683 raft_consensus.cc:729] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.796449 32683 consensus_queue.cc:260] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.796646 32683 ts_tablet_manager.cc:1398] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:38.798019 32186 raft_consensus.cc:482] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.798099 32186 raft_consensus.cc:504] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:38.798452 32186 leader_election.cc:288] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:38.798803 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71241540f585474a8f62da16ed2d9874" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:38.798804 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71241540f585474a8f62da16ed2d9874" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" is_pre_election: true
I20240501 14:09:38.798910 31320 raft_consensus.cc:2454] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:38.798928 31580 raft_consensus.cc:2454] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:38.799160 31125 leader_election.cc:302] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 30e0b8849e3841ca9ab0de01e719ee4a; no voters:
I20240501 14:09:38.799333 32186 raft_consensus.cc:2781] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.799386 32186 raft_consensus.cc:482] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.799423 32186 raft_consensus.cc:3037] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.800374 32186 raft_consensus.cc:504] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:38.800660 32186 leader_election.cc:288] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Requested vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:38.800959 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71241540f585474a8f62da16ed2d9874" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:38.801087 31320 raft_consensus.cc:3037] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.801065 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71241540f585474a8f62da16ed2d9874" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:38.801173 31580 raft_consensus.cc:3037] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.801775 31320 raft_consensus.cc:2454] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:38.801874 31580 raft_consensus.cc:2454] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:38.802021 31125 leader_election.cc:302] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 30e0b8849e3841ca9ab0de01e719ee4a; no voters:
I20240501 14:09:38.802175 32186 raft_consensus.cc:2781] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.802258 32186 raft_consensus.cc:686] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Becoming Leader. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Running, Role: LEADER
I20240501 14:09:38.802376 32186 consensus_queue.cc:237] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:38.803869 31046 catalog_manager.cc:5484] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.817189 32313 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.817301 32313 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.817677 32313 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:38.818087 31449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:38.818068 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" is_pre_election: true
I20240501 14:09:38.818225 31320 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:38.818225 31449 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:38.818493 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:38.818663 32313 raft_consensus.cc:2781] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.818720 32313 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.818748 32313 raft_consensus.cc:3037] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.819583 32313 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.819885 32313 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Requested vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:38.820184 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:38.820300 31320 raft_consensus.cc:3037] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.820322 31449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:38.820442 31449 raft_consensus.cc:3037] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.821264 31320 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:38.821357 31449 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:38.821554 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:38.821764 32313 raft_consensus.cc:2781] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.821843 32313 raft_consensus.cc:686] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Becoming Leader. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Running, Role: LEADER
I20240501 14:09:38.821982 32313 consensus_queue.cc:237] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.823518 31046 catalog_manager.cc:5484] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: term changed from 0 to 1, leader changed from <none> to e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68). New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.848124 32313 raft_consensus.cc:482] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.848232 32313 raft_consensus.cc:504] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.848613 32313 leader_election.cc:288] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:38.848961 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043344ce36a74a18a0d67ff5bd473e68" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:09:38.849095 31187 raft_consensus.cc:2454] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:38.849097 31449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043344ce36a74a18a0d67ff5bd473e68" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:38.849220 31449 raft_consensus.cc:2454] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:38.849462 31515 leader_election.cc:302] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:38.849697 32313 raft_consensus.cc:2781] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.849743 32313 raft_consensus.cc:482] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.849778 32313 raft_consensus.cc:3037] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.850531 32313 raft_consensus.cc:504] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.850842 32313 leader_election.cc:288] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Requested vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:38.851145 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043344ce36a74a18a0d67ff5bd473e68" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:38.851148 31449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043344ce36a74a18a0d67ff5bd473e68" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:38.851248 31187 raft_consensus.cc:3037] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.851274 31449 raft_consensus.cc:3037] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.851969 31449 raft_consensus.cc:2454] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:38.852133 31187 raft_consensus.cc:2454] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:38.852236 31516 leader_election.cc:302] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a6fcc099d1340c383e5a429643d5b7c, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:38.852438 32313 raft_consensus.cc:2781] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.852525 32313 raft_consensus.cc:686] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Becoming Leader. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Running, Role: LEADER
I20240501 14:09:38.852660 32313 consensus_queue.cc:237] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } }
I20240501 14:09:38.854338 31046 catalog_manager.cc:5484] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: term changed from 0 to 1, leader changed from <none> to e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68). New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:38.968521 32697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:38.968739 32697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:38.971750 32697 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 14:09:39.065084 31187 raft_consensus.cc:1261] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.065536 31449 raft_consensus.cc:1261] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.065523 32313 consensus_queue.cc:1035] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.066124 32313 consensus_queue.cc:1035] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.072170 31451 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.072170 31319 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.072561 32365 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.073092 32365 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.073505 31320 raft_consensus.cc:1261] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.074024 31580 raft_consensus.cc:1261] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.076661 32186 consensus_queue.cc:1035] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.080523 32526 consensus_queue.cc:1035] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.695 ms
time per row: 0.0568477 ms
I20240501 14:09:39.131533 31043 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52370:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:39.136638 31169 tablet_service.cc:1467] Processing CreateTablet for tablet 5d994081c3414f779ba2cae431cb1d97 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b01dc7a764a349c9a4a9180993e4fbc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.136886 31560 tablet_service.cc:1467] Processing CreateTablet for tablet 5d994081c3414f779ba2cae431cb1d97 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b01dc7a764a349c9a4a9180993e4fbc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.137017 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d994081c3414f779ba2cae431cb1d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.137120 31560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d994081c3414f779ba2cae431cb1d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.137146 31429 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8bd03c05d84fdaaf138a6d0608e30e (DEFAULT_TABLE table=rebalance_test_table_8 [id=b01dc7a764a349c9a4a9180993e4fbc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.137419 31429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8bd03c05d84fdaaf138a6d0608e30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.137418 31299 tablet_service.cc:1467] Processing CreateTablet for tablet 5d994081c3414f779ba2cae431cb1d97 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b01dc7a764a349c9a4a9180993e4fbc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.137651 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d994081c3414f779ba2cae431cb1d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.139084 32683 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:39.139472 32685 tablet_bootstrap.cc:492] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:39.139505 32681 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:39.139868 32683 tablet_bootstrap.cc:654] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.140203 31299 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8bd03c05d84fdaaf138a6d0608e30e (DEFAULT_TABLE table=rebalance_test_table_8 [id=b01dc7a764a349c9a4a9180993e4fbc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.140285 32685 tablet_bootstrap.cc:654] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.140329 32681 tablet_bootstrap.cc:654] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.140416 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8bd03c05d84fdaaf138a6d0608e30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.140874 32682 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:39.141417 31168 tablet_service.cc:1467] Processing CreateTablet for tablet d265eff200b645bfac8f8906654385f6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b01dc7a764a349c9a4a9180993e4fbc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.141631 31168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d265eff200b645bfac8f8906654385f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.141661 32682 tablet_bootstrap.cc:654] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.142148 31560 tablet_service.cc:1467] Processing CreateTablet for tablet d265eff200b645bfac8f8906654385f6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b01dc7a764a349c9a4a9180993e4fbc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.142377 31560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d265eff200b645bfac8f8906654385f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.143572 31429 tablet_service.cc:1467] Processing CreateTablet for tablet d265eff200b645bfac8f8906654385f6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b01dc7a764a349c9a4a9180993e4fbc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.143803 31429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d265eff200b645bfac8f8906654385f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.143914 32681 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:39.144003 32681 ts_tablet_manager.cc:1367] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240501 14:09:39.144434 32681 raft_consensus.cc:348] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:39.144536 32681 raft_consensus.cc:374] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.144567 32681 raft_consensus.cc:729] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.144685 32681 consensus_queue.cc:260] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:39.144925 32682 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:39.145004 32682 ts_tablet_manager.cc:1367] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:39.145010 32681 ts_tablet_manager.cc:1398] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:09:39.145165 32681 tablet_bootstrap.cc:492] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:39.145396 32682 raft_consensus.cc:348] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:39.145514 32682 raft_consensus.cc:374] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.145550 32682 raft_consensus.cc:729] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.145655 32682 consensus_queue.cc:260] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:39.145785 31169 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8bd03c05d84fdaaf138a6d0608e30e (DEFAULT_TABLE table=rebalance_test_table_8 [id=b01dc7a764a349c9a4a9180993e4fbc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.145892 32682 ts_tablet_manager.cc:1398] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.145996 32681 tablet_bootstrap.cc:654] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.146020 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8bd03c05d84fdaaf138a6d0608e30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.146054 32682 tablet_bootstrap.cc:492] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:39.147089 32682 tablet_bootstrap.cc:654] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.148097 32682 tablet_bootstrap.cc:492] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:39.148185 32682 ts_tablet_manager.cc:1367] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:39.148286 32681 tablet_bootstrap.cc:492] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:39.148371 32681 ts_tablet_manager.cc:1367] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:39.148595 32682 raft_consensus.cc:348] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:39.148700 32682 raft_consensus.cc:374] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.148742 32682 raft_consensus.cc:729] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.148783 32681 raft_consensus.cc:348] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:39.148921 32681 raft_consensus.cc:374] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.148903 32682 consensus_queue.cc:260] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:39.148965 32681 raft_consensus.cc:729] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.149089 32681 consensus_queue.cc:260] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:39.149165 32682 ts_tablet_manager.cc:1398] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.149442 32681 ts_tablet_manager.cc:1398] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.150853 32526 raft_consensus.cc:482] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:39.150936 32526 raft_consensus.cc:504] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:39.151250 32526 leader_election.cc:288] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:39.151562 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d265eff200b645bfac8f8906654385f6" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:39.151938 32682 tablet_bootstrap.cc:492] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:39.152057 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d265eff200b645bfac8f8906654385f6" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
W20240501 14:09:39.152494 31124 leader_election.cc:341] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:09:39.152628 31123 leader_election.cc:341] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:39.152704 31123 leader_election.cc:302] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: 6a6fcc099d1340c383e5a429643d5b7c, e7eada5032fb45c3a5e3196e2b3faaab
I20240501 14:09:39.152786 32682 tablet_bootstrap.cc:654] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.152863 32526 raft_consensus.cc:2726] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:39.153816 32682 tablet_bootstrap.cc:492] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:39.153890 32682 ts_tablet_manager.cc:1367] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:39.154246 32682 raft_consensus.cc:348] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:39.154326 32682 raft_consensus.cc:374] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.154354 32682 raft_consensus.cc:729] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.154457 32682 consensus_queue.cc:260] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:39.154723 32682 ts_tablet_manager.cc:1398] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.155282 32685 tablet_bootstrap.cc:492] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:39.155376 32685 ts_tablet_manager.cc:1367] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:39.155460 32683 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:39.155543 32683 ts_tablet_manager.cc:1367] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:39.155802 32685 raft_consensus.cc:348] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:39.155889 32685 raft_consensus.cc:374] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.155915 32685 raft_consensus.cc:729] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.155964 32683 raft_consensus.cc:348] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:39.156070 32683 raft_consensus.cc:374] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.156033 32685 consensus_queue.cc:260] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:39.156111 32683 raft_consensus.cc:729] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.156224 32683 consensus_queue.cc:260] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:39.156313 32685 ts_tablet_manager.cc:1398] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.156472 32685 tablet_bootstrap.cc:492] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:39.156500 32683 ts_tablet_manager.cc:1398] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:39.156627 32683 tablet_bootstrap.cc:492] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:39.157371 32685 tablet_bootstrap.cc:654] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.157415 32683 tablet_bootstrap.cc:654] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.158514 32685 tablet_bootstrap.cc:492] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:39.158596 32685 ts_tablet_manager.cc:1367] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:39.159049 32683 tablet_bootstrap.cc:492] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:39.159020 32685 raft_consensus.cc:348] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:39.159113 32685 raft_consensus.cc:374] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.159150 32685 raft_consensus.cc:729] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.159263 32683 ts_tablet_manager.cc:1367] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:39.159291 32685 consensus_queue.cc:260] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:39.159628 32685 ts_tablet_manager.cc:1398] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.160444 32683 raft_consensus.cc:348] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:39.160593 32683 raft_consensus.cc:374] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.160643 32683 raft_consensus.cc:729] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.160764 32683 consensus_queue.cc:260] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:39.161195 32683 ts_tablet_manager.cc:1398] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:39.183657 32536 raft_consensus.cc:482] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:39.183773 32536 raft_consensus.cc:504] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:39.184152 32536 leader_election.cc:288] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:39.184453 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d994081c3414f779ba2cae431cb1d97" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:09:39.184500 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d994081c3414f779ba2cae431cb1d97" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:39.184583 31187 raft_consensus.cc:2454] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:39.184640 31580 raft_consensus.cc:2454] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:39.184934 31255 leader_election.cc:302] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:39.185096 32536 raft_consensus.cc:2781] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:39.185153 32536 raft_consensus.cc:482] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:39.185199 32536 raft_consensus.cc:3037] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.186067 32536 raft_consensus.cc:504] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:39.186376 32536 leader_election.cc:288] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Requested vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:39.186635 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d994081c3414f779ba2cae431cb1d97" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:39.186697 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d994081c3414f779ba2cae431cb1d97" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:39.186750 31187 raft_consensus.cc:3037] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.186796 31580 raft_consensus.cc:3037] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.187649 31187 raft_consensus.cc:2454] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:39.187649 31580 raft_consensus.cc:2454] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:39.187947 31255 leader_election.cc:302] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:39.188144 32536 raft_consensus.cc:2781] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:39.188233 32536 raft_consensus.cc:686] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Becoming Leader. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Running, Role: LEADER
I20240501 14:09:39.188361 32536 consensus_queue.cc:237] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:39.190090 31040 catalog_manager.cc:5484] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66). New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:39.296227 32526 raft_consensus.cc:482] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:39.296355 32526 raft_consensus.cc:504] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:39.296747 32526 leader_election.cc:288] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:39.297139 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8bd03c05d84fdaaf138a6d0608e30e" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:39.297156 31319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8bd03c05d84fdaaf138a6d0608e30e" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" is_pre_election: true
I20240501 14:09:39.297278 31451 raft_consensus.cc:2454] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:39.297291 31319 raft_consensus.cc:2454] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:39.297556 31123 leader_election.cc:302] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:39.297716 32526 raft_consensus.cc:2781] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:39.297775 32526 raft_consensus.cc:482] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:39.297816 32526 raft_consensus.cc:3037] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.298568 32526 raft_consensus.cc:504] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:39.298863 32526 leader_election.cc:288] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Requested vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:39.299163 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8bd03c05d84fdaaf138a6d0608e30e" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:39.299175 31319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8bd03c05d84fdaaf138a6d0608e30e" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:39.299268 31319 raft_consensus.cc:3037] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.299296 31451 raft_consensus.cc:3037] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.299973 31319 raft_consensus.cc:2454] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:39.300055 31451 raft_consensus.cc:2454] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:39.300268 31125 leader_election.cc:302] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 30e0b8849e3841ca9ab0de01e719ee4a; no voters:
I20240501 14:09:39.300447 32526 raft_consensus.cc:2781] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:39.300537 32526 raft_consensus.cc:686] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Becoming Leader. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Running, Role: LEADER
I20240501 14:09:39.300666 32526 consensus_queue.cc:237] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:39.302390 31040 catalog_manager.cc:5484] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:39.567994 32536 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.571081 32430 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.631057 32731 raft_consensus.cc:482] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:39.631196 32731 raft_consensus.cc:504] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:39.631628 32731 leader_election.cc:288] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:39.632068 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d265eff200b645bfac8f8906654385f6" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:39.632091 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d265eff200b645bfac8f8906654385f6" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:09:39.632216 31187 raft_consensus.cc:2454] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:39.632227 31451 raft_consensus.cc:2454] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:39.632551 31515 leader_election.cc:302] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:39.632714 32731 raft_consensus.cc:2781] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:39.632786 32731 raft_consensus.cc:482] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:39.632844 32731 raft_consensus.cc:3037] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.633652 32731 raft_consensus.cc:504] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:39.633960 32731 leader_election.cc:288] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Requested vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:39.634274 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d265eff200b645bfac8f8906654385f6" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:39.634268 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d265eff200b645bfac8f8906654385f6" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:39.634389 31451 raft_consensus.cc:3037] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.634389 31187 raft_consensus.cc:3037] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.635293 31187 raft_consensus.cc:2454] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:39.635293 31451 raft_consensus.cc:2454] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:39.635574 31515 leader_election.cc:302] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:39.635735 32731 raft_consensus.cc:2781] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:39.635893 32731 raft_consensus.cc:686] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Becoming Leader. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Running, Role: LEADER
I20240501 14:09:39.636022 32731 consensus_queue.cc:237] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } }
I20240501 14:09:39.637773 31043 catalog_manager.cc:5484] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: term changed from 0 to 1, leader changed from <none> to e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68). New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:39.697121 32653 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.700665 32653 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:39.814797 32752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:39.815045 32752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:39.818569 32752 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 14:09:39.935362 31451 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.937021 32731 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.938280 31189 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.941416 32731 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 141.003 ms
time per row: 0.0705013 ms
I20240501 14:09:40.006762 31046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52370:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:40.012814 31299 tablet_service.cc:1467] Processing CreateTablet for tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.012889 31429 tablet_service.cc:1467] Processing CreateTablet for tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.013190 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.013190 31429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.015239 337 tablet_bootstrap.cc:492] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:40.015580 31429 tablet_service.cc:1467] Processing CreateTablet for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.015801 31429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.015956 31298 tablet_service.cc:1467] Processing CreateTablet for tablet b1006490ec2841b2a3365295caaf19e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.016171 31298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1006490ec2841b2a3365295caaf19e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.016269 31169 tablet_service.cc:1467] Processing CreateTablet for tablet b1006490ec2841b2a3365295caaf19e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.016515 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1006490ec2841b2a3365295caaf19e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.018543 337 tablet_bootstrap.cc:654] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.018635 338 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:40.019495 338 tablet_bootstrap.cc:654] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.019477 31561 tablet_service.cc:1467] Processing CreateTablet for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.019477 31560 tablet_service.cc:1467] Processing CreateTablet for tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.019762 31561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.019913 31560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.020130 31562 tablet_service.cc:1467] Processing CreateTablet for tablet b1006490ec2841b2a3365295caaf19e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.020310 31562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1006490ec2841b2a3365295caaf19e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.021556 341 tablet_bootstrap.cc:492] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:40.022384 341 tablet_bootstrap.cc:654] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.023375 341 tablet_bootstrap.cc:492] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:40.023453 341 ts_tablet_manager.cc:1367] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:40.023823 341 raft_consensus.cc:348] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.023841 31299 tablet_service.cc:1467] Processing CreateTablet for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.023919 341 raft_consensus.cc:374] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.023952 341 raft_consensus.cc:729] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.024058 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.024086 341 consensus_queue.cc:260] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.024439 341 ts_tablet_manager.cc:1398] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.024649 341 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:40.025467 341 tablet_bootstrap.cc:654] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.026088 336 tablet_bootstrap.cc:492] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:40.026464 341 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:40.026542 341 ts_tablet_manager.cc:1367] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:40.026834 336 tablet_bootstrap.cc:654] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.026934 341 raft_consensus.cc:348] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:40.027030 341 raft_consensus.cc:374] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.027058 341 raft_consensus.cc:729] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.027173 341 consensus_queue.cc:260] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:40.027427 341 ts_tablet_manager.cc:1398] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.027539 337 tablet_bootstrap.cc:492] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:40.027565 341 tablet_bootstrap.cc:492] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:40.027622 337 ts_tablet_manager.cc:1367] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240501 14:09:40.027972 336 tablet_bootstrap.cc:492] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:40.028054 336 ts_tablet_manager.cc:1367] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:40.028012 337 raft_consensus.cc:348] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.028141 337 raft_consensus.cc:374] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.028174 337 raft_consensus.cc:729] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.028306 337 consensus_queue.cc:260] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.028430 336 raft_consensus.cc:348] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.028514 336 raft_consensus.cc:374] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.028538 336 raft_consensus.cc:729] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.028564 337 ts_tablet_manager.cc:1398] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.028632 336 consensus_queue.cc:260] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.028734 337 tablet_bootstrap.cc:492] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:40.028754 341 tablet_bootstrap.cc:654] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.028930 336 ts_tablet_manager.cc:1398] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.029079 336 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:40.029517 337 tablet_bootstrap.cc:654] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.029843 336 tablet_bootstrap.cc:654] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.030819 336 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:40.030897 336 ts_tablet_manager.cc:1367] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:40.031299 336 raft_consensus.cc:348] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:40.031396 336 raft_consensus.cc:374] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.031428 336 raft_consensus.cc:729] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.031533 337 tablet_bootstrap.cc:492] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c: No bootstrap required, opened a new log
I20240501 14:09:40.031530 336 consensus_queue.cc:260] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:40.031600 337 ts_tablet_manager.cc:1367] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:40.031777 336 ts_tablet_manager.cc:1398] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:40.031910 336 tablet_bootstrap.cc:492] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:40.032023 337 raft_consensus.cc:348] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.032112 337 raft_consensus.cc:374] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.032141 337 raft_consensus.cc:729] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.032449 337 consensus_queue.cc:260] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.032856 337 ts_tablet_manager.cc:1398] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.032902 336 tablet_bootstrap.cc:654] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.034168 341 tablet_bootstrap.cc:492] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab: No bootstrap required, opened a new log
I20240501 14:09:40.034245 341 ts_tablet_manager.cc:1367] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:40.034651 341 raft_consensus.cc:348] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.034746 341 raft_consensus.cc:374] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.034780 341 raft_consensus.cc:729] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.034883 341 consensus_queue.cc:260] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.035140 341 ts_tablet_manager.cc:1398] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.037498 338 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1: No bootstrap required, opened a new log
I20240501 14:09:40.037542 336 tablet_bootstrap.cc:492] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a: No bootstrap required, opened a new log
I20240501 14:09:40.037593 338 ts_tablet_manager.cc:1367] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.000s
I20240501 14:09:40.037611 336 ts_tablet_manager.cc:1367] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240501 14:09:40.037968 336 raft_consensus.cc:348] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.038048 336 raft_consensus.cc:374] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.038077 336 raft_consensus.cc:729] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.038033 338 raft_consensus.cc:348] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:40.038143 338 raft_consensus.cc:374] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.038185 338 raft_consensus.cc:729] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.038184 336 consensus_queue.cc:260] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.038296 338 consensus_queue.cc:260] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:40.038421 336 ts_tablet_manager.cc:1398] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:40.038547 338 ts_tablet_manager.cc:1398] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.112535 32745 raft_consensus.cc:482] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:40.112663 32745 raft_consensus.cc:504] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.113076 32745 leader_election.cc:288] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:40.113477 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4603d1b1b1d441b1b0be0464e2f2c9e2" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:40.113492 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4603d1b1b1d441b1b0be0464e2f2c9e2" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:40.113608 31581 raft_consensus.cc:2454] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:40.113617 31451 raft_consensus.cc:2454] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 0.
I20240501 14:09:40.113937 31255 leader_election.cc:302] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:40.114100 32745 raft_consensus.cc:2781] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:40.114161 32745 raft_consensus.cc:482] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:40.114187 32745 raft_consensus.cc:3037] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.115016 32745 raft_consensus.cc:504] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.115320 32745 leader_election.cc:288] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Requested vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:40.115643 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4603d1b1b1d441b1b0be0464e2f2c9e2" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:40.115768 31451 raft_consensus.cc:3037] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.115777 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4603d1b1b1d441b1b0be0464e2f2c9e2" candidate_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:40.115890 31581 raft_consensus.cc:3037] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.116726 31451 raft_consensus.cc:2454] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:40.116742 31581 raft_consensus.cc:2454] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e0b8849e3841ca9ab0de01e719ee4a in term 1.
I20240501 14:09:40.117013 31254 leader_election.cc:302] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e0b8849e3841ca9ab0de01e719ee4a, 6a6fcc099d1340c383e5a429643d5b7c; no voters:
I20240501 14:09:40.117175 32745 raft_consensus.cc:2781] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:40.117252 32745 raft_consensus.cc:686] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Becoming Leader. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Running, Role: LEADER
I20240501 14:09:40.117412 32745 consensus_queue.cc:237] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.119220 31046 catalog_manager.cc:5484] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66). New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:40.180869 32526 raft_consensus.cc:482] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:40.180991 32526 raft_consensus.cc:504] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:40.181363 32526 leader_election.cc:288] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:40.181633 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1006490ec2841b2a3365295caaf19e1" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:40.181696 31319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1006490ec2841b2a3365295caaf19e1" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" is_pre_election: true
I20240501 14:09:40.181759 31581 raft_consensus.cc:2454] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:40.181861 31319 raft_consensus.cc:2454] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 0.
I20240501 14:09:40.182053 31124 leader_election.cc:302] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:40.182206 32526 raft_consensus.cc:2781] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:40.182265 32526 raft_consensus.cc:482] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:40.182308 32526 raft_consensus.cc:3037] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.183085 32526 raft_consensus.cc:504] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:40.183401 32526 leader_election.cc:288] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Requested vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:40.183705 31319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1006490ec2841b2a3365295caaf19e1" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:40.183689 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1006490ec2841b2a3365295caaf19e1" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:40.183810 31581 raft_consensus.cc:3037] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.183810 31319 raft_consensus.cc:3037] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.184720 31319 raft_consensus.cc:2454] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:40.184720 31581 raft_consensus.cc:2454] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 1.
I20240501 14:09:40.185034 31124 leader_election.cc:302] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:40.185216 32526 raft_consensus.cc:2781] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:40.185299 32526 raft_consensus.cc:686] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Becoming Leader. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Running, Role: LEADER
I20240501 14:09:40.185444 32526 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } }
I20240501 14:09:40.187183 31046 catalog_manager.cc:5484] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65). New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:40.195698 335 raft_consensus.cc:482] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:40.195830 335 raft_consensus.cc:504] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.196245 335 leader_election.cc:288] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:40.196650 31319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b83f4cd48b4b4251a8ac4ddebdb8e4e1" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" is_pre_election: true
I20240501 14:09:40.196646 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b83f4cd48b4b4251a8ac4ddebdb8e4e1" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c" is_pre_election: true
I20240501 14:09:40.196795 31451 raft_consensus.cc:2454] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:40.196815 31319 raft_consensus.cc:2454] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 0.
I20240501 14:09:40.197134 31516 leader_election.cc:302] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a6fcc099d1340c383e5a429643d5b7c, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:40.197309 335 raft_consensus.cc:2781] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:40.197367 335 raft_consensus.cc:482] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:40.197402 335 raft_consensus.cc:3037] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.198225 335 raft_consensus.cc:504] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.198534 335 leader_election.cc:288] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Requested vote from peers 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413), 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:40.198825 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b83f4cd48b4b4251a8ac4ddebdb8e4e1" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:40.198874 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b83f4cd48b4b4251a8ac4ddebdb8e4e1" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
I20240501 14:09:40.198926 31320 raft_consensus.cc:3037] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.198980 31451 raft_consensus.cc:3037] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.199673 31451 raft_consensus.cc:2454] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:40.199681 31320 raft_consensus.cc:2454] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 1.
I20240501 14:09:40.199920 31516 leader_election.cc:302] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a6fcc099d1340c383e5a429643d5b7c, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:40.200085 335 raft_consensus.cc:2781] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:40.200177 335 raft_consensus.cc:686] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Becoming Leader. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Running, Role: LEADER
I20240501 14:09:40.200297 335 consensus_queue.cc:237] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } }
I20240501 14:09:40.202154 31046 catalog_manager.cc:5484] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: term changed from 0 to 1, leader changed from <none> to e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68). New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:40.320386 352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:40.320638 352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:40.323921 352 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 14:09:40.429656 31320 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.430037 31451 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.430568 32731 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.431269 32731 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.432368 31449 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.433096 32745 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.433539 31319 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.433674 31581 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.433674 31580 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.434808 32526 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.435106 32526 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.442472 32745 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.465898 383 mvcc.cc:205] Tried to move back new op lower bound from 7022889289428242432 to 7022889288501501952. Current Snapshot: MvccSnapshot[applied={T|T < 7022889289428242432}]
INSERT report
rows total: 2000
time total: 129.948 ms
time per row: 0.0649738 ms
I20240501 14:09:40.499809 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.69:45429
--local_ip_for_outbound_sockets=127.23.191.69
--tserver_master_addrs=127.23.191.126:41449
--webserver_port=44577
--webserver_interface=127.23.191.69
--builtin_ntp_servers=127.23.191.84:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:40.608902 393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:40.609180 393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:40.609246 393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:40.613263 393 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:40.613346 393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:40.613438 393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.69
I20240501 14:09:40.617410 393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.69:45429
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.23.191.69
--webserver_port=44577
--tserver_master_addrs=127.23.191.126:41449
--never_fsync=true
--heap_profile_path=/tmp/kudu.393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.69
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:40.618336 393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:40.619675 393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:40.628545 398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:40.628868 399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:40.629555 393 server_base.cc:1034] running on GCE node
W20240501 14:09:40.629827 401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:40.630228 393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:40.630792 393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:40.631928 393 hybrid_clock.cc:648] HybridClock initialized: now 1714572580631916 us; error 27 us; skew 500 ppm
I20240501 14:09:40.633934 393 webserver.cc:460] Webserver started at http://127.23.191.69:44577/ using document root <none> and password file <none>
I20240501 14:09:40.634497 393 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:40.634572 393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:40.638298 393 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20240501 14:09:40.640113 407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:40.641211 393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:09:40.641320 393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b934fd09590a40ada094b9715d990a00"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:40.641816 393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:40.665055 393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:40.665666 393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:40.665841 393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:40.666381 393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:40.667246 393 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:40.667299 393 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:40.667351 393 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:40.667395 393 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:40.675577 393 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.69:45429
I20240501 14:09:40.675608 520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.69:45429 every 8 connection(s)
I20240501 14:09:40.676589 393 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:09:40.684922 521 heartbeater.cc:347] Connected to a master server at 127.23.191.126:41449
I20240501 14:09:40.685148 521 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:40.685487 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 393
I20240501 14:09:40.685701 521 heartbeater.cc:511] Master 127.23.191.126:41449 requested a full tablet report, sending...
I20240501 14:09:40.685889 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.70:38181
--local_ip_for_outbound_sockets=127.23.191.70
--tserver_master_addrs=127.23.191.126:41449
--webserver_port=41227
--webserver_interface=127.23.191.70
--builtin_ntp_servers=127.23.191.84:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:40.686825 31046 ts_manager.cc:194] Re-registered known tserver with Master: b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429)
I20240501 14:09:40.687404 31046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.69:46723
W20240501 14:09:40.795027 524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:40.795255 524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:40.795297 524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:40.798383 524 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:40.798460 524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:40.798529 524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.70
I20240501 14:09:40.802388 524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.70:38181
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.23.191.70
--webserver_port=41227
--tserver_master_addrs=127.23.191.126:41449
--never_fsync=true
--heap_profile_path=/tmp/kudu.524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.70
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:40.803269 524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:40.804358 524 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:40.812791 529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:40.813282 530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:40.813851 532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:40.814405 524 server_base.cc:1034] running on GCE node
I20240501 14:09:40.814797 524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:40.815346 524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:40.816502 524 hybrid_clock.cc:648] HybridClock initialized: now 1714572580816488 us; error 26 us; skew 500 ppm
I20240501 14:09:40.818370 524 webserver.cc:460] Webserver started at http://127.23.191.70:41227/ using document root <none> and password file <none>
I20240501 14:09:40.818944 524 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:40.819026 524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:40.822943 524 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:40.824993 538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:40.826015 524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:40.826121 524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9438789dc61f4501ad1d4ceaeccee458"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:40.826553 524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:40.849802 524 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:40.850390 524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:40.850553 524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:40.851085 524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:40.851979 524 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:40.852025 524 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:40.852073 524 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:40.852102 524 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:40.860226 524 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.70:38181
I20240501 14:09:40.860245 651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.70:38181 every 8 connection(s)
I20240501 14:09:40.861275 524 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:09:40.862004 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 524
I20240501 14:09:40.862388 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.71:35407
--local_ip_for_outbound_sockets=127.23.191.71
--tserver_master_addrs=127.23.191.126:41449
--webserver_port=36285
--webserver_interface=127.23.191.71
--builtin_ntp_servers=127.23.191.84:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:40.870633 652 heartbeater.cc:347] Connected to a master server at 127.23.191.126:41449
I20240501 14:09:40.870852 652 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:40.871352 652 heartbeater.cc:511] Master 127.23.191.126:41449 requested a full tablet report, sending...
I20240501 14:09:40.872395 31046 ts_manager.cc:194] Re-registered known tserver with Master: 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:09:40.873054 31046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.70:56375
W20240501 14:09:40.973646 655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:40.973888 655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:40.973961 655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:40.977337 655 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:40.977412 655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:40.977483 655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.71
I20240501 14:09:40.981197 655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.71:35407
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.23.191.71
--webserver_port=36285
--tserver_master_addrs=127.23.191.126:41449
--never_fsync=true
--heap_profile_path=/tmp/kudu.655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.71
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:40.982080 655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:40.983268 655 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:40.991842 661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:40.992168 660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:40.992727 663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:40.993218 655 server_base.cc:1034] running on GCE node
I20240501 14:09:40.993629 655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:40.994186 655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:40.995342 655 hybrid_clock.cc:648] HybridClock initialized: now 1714572580995325 us; error 28 us; skew 500 ppm
I20240501 14:09:40.997236 655 webserver.cc:460] Webserver started at http://127.23.191.71:36285/ using document root <none> and password file <none>
I20240501 14:09:40.997759 655 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:40.997828 655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:41.001382 655 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:41.003204 669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:41.004177 655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:41.004277 655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a0caf2974041408bb32362bfc4284eaa"
format_stamp: "Formatted at 2024-05-01 14:09:30 on dist-test-slave-c4g3"
I20240501 14:09:41.004698 655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:41.021322 655 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:41.021879 655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:41.022033 655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:41.022547 655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:41.023424 655 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:41.023471 655 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:41.023524 655 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:41.023548 655 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:41.033022 655 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.71:35407
I20240501 14:09:41.033061 782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.71:35407 every 8 connection(s)
I20240501 14:09:41.034041 655 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:09:41.039072 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 655
I20240501 14:09:41.039444 24317 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
/tmp/dist-test-tasklfWEhd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.191.72:39799
--local_ip_for_outbound_sockets=127.23.191.72
--tserver_master_addrs=127.23.191.126:41449
--webserver_port=43401
--webserver_interface=127.23.191.72
--builtin_ntp_servers=127.23.191.84:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:41.043210 783 heartbeater.cc:347] Connected to a master server at 127.23.191.126:41449
I20240501 14:09:41.043483 783 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:41.044144 783 heartbeater.cc:511] Master 127.23.191.126:41449 requested a full tablet report, sending...
I20240501 14:09:41.045262 31046 ts_manager.cc:194] Re-registered known tserver with Master: a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
I20240501 14:09:41.046031 31046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.71:50095
W20240501 14:09:41.148269 786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:41.148483 786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:41.148526 786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:41.151547 786 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:41.151620 786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:41.151686 786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.191.72
I20240501 14:09:41.155268 786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.191.84:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.191.72:39799
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.23.191.72
--webserver_port=43401
--tserver_master_addrs=127.23.191.126:41449
--never_fsync=true
--heap_profile_path=/tmp/kudu.786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.191.72
--log_dir=/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:41.156134 786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:41.157320 786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:41.165798 792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:41.166139 791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:41.166839 794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:41.167340 786 server_base.cc:1034] running on GCE node
I20240501 14:09:41.167742 786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:41.168359 786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:41.169520 786 hybrid_clock.cc:648] HybridClock initialized: now 1714572581169495 us; error 40 us; skew 500 ppm
I20240501 14:09:41.171389 786 webserver.cc:460] Webserver started at http://127.23.191.72:43401/ using document root <none> and password file <none>
I20240501 14:09:41.171919 786 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:41.171986 786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:41.175606 786 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:41.177455 800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:41.178426 786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:41.178537 786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "a63e0dd1ba2e4a8db8ace9718a700c32"
format_stamp: "Formatted at 2024-05-01 14:09:31 on dist-test-slave-c4g3"
I20240501 14:09:41.178936 786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:41.190507 786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:41.191159 786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:41.191325 786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:41.191829 786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:41.192785 786 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:41.192854 786 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:41.192919 786 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:41.192945 786 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:41.202359 786 rpc_server.cc:307] RPC server started. Bound to: 127.23.191.72:39799
I20240501 14:09:41.202412 913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.191.72:39799 every 8 connection(s)
I20240501 14:09:41.203375 786 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:09:41.205017 24317 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu as pid 786
I20240501 14:09:41.214481 914 heartbeater.cc:347] Connected to a master server at 127.23.191.126:41449
I20240501 14:09:41.214735 914 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:41.215405 914 heartbeater.cc:511] Master 127.23.191.126:41449 requested a full tablet report, sending...
I20240501 14:09:41.216454 31046 ts_manager.cc:194] Re-registered known tserver with Master: a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799)
I20240501 14:09:41.217124 31046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.191.72:55421
I20240501 14:09:41.550029 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:41.573943 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.577713 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.589430 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.598838 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.613165 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:41.619371 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:41.639817 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.655685 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.657505 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:41.660451 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.665503 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:41.669457 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:41.672675 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.675881 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.677762 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:41.684482 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.686568 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.689555 521 heartbeater.cc:503] Master 127.23.191.126:41449 was elected leader, sending a full tablet report...
I20240501 14:09:41.691287 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.692684 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.704620 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.729964 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:41.737859 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.770887 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:41.859782 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:41.869016 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.875190 652 heartbeater.cc:503] Master 127.23.191.126:41449 was elected leader, sending a full tablet report...
I20240501 14:09:41.887537 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.887848 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:41.887992 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:41.893712 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.901923 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:41.902096 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.906919 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:41.915552 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.915949 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.921257 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:41.921572 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.922508 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.923472 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:41.949779 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.049048 783 heartbeater.cc:503] Master 127.23.191.126:41449 was elected leader, sending a full tablet report...
I20240501 14:09:42.063153 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.066627 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.072947 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.074755 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.086966 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.087582 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.095211 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:42.123461 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.168816 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.173256 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.174738 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.175942 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.179411 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.184695 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.185565 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.205082 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.209761 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.210600 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:42.215018 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.218865 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.219583 914 heartbeater.cc:503] Master 127.23.191.126:41449 was elected leader, sending a full tablet report...
I20240501 14:09:42.224597 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.231858 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:42.246121 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.266029 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.373981 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.385530 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.385533 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.391669 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.401521 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.402446 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.406814 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.413911 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.424613 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.427523 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.432982 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:42.433921 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.434988 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.465037 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:42.472306 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.475615 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.538703 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.544484 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.548300 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.549757 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.551795 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.561195 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.568153 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.569173 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:42.616169 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.618062 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.626025 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.638250 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.641436 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.650846 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.677332 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.680732 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:42.686631 31320 consensus_queue.cc:237] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:42.688271 31581 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.688669 1219 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.690755 31187 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.691147 1219 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.697582 1555 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:42.703128 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.702947 31581 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:42.706025 31039 catalog_manager.cc:5484] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:42.714311 31581 consensus_queue.cc:237] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:42.715862 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.716455 31449 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.716732 31319 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.716936 1214 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.717571 1214 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:42.723698 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d2c173b2c32494eb16e46a51ca36e31. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.724772 1214 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:42.728320 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.730441 31449 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:42.734174 31040 catalog_manager.cc:5484] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:42.736819 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.749136 31320 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:42.753399 31187 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:42.761790 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.762821 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.768571 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.772254 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:42.782399 31449 consensus_queue.cc:237] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:42.784363 31187 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.784567 31320 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.784914 382 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.785243 382 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:42.787541 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f91909d3b1b46e9b858ce5c5b2e7975. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.788089 382 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:42.790027 31187 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
W20240501 14:09:42.790186 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a542c07a3f841b0a31051a8a37083bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.794332 31320 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:42.819586 31045 catalog_manager.cc:5484] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:42.824303 31580 consensus_queue.cc:237] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:42.825875 31449 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.826272 1589 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:42.827986 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b83f4cd48b4b4251a8ac4ddebdb8e4e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.828603 31320 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.829072 1589 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.831331 1580 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:42.833673 31320 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:42.834668 31039 catalog_manager.cc:5484] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:42.835476 31449 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:42.845068 31320 consensus_queue.cc:237] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:42.846923 31449 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.847290 1219 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.848235 31187 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.849130 1219 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:42.851260 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0399a419a25c4981a2ee90d76cbf59b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.851781 1219 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:42.852212 31449 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:42.853893 31187 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:42.855624 31043 catalog_manager.cc:5484] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:42.860440 31187 consensus_queue.cc:237] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:42.861853 31320 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.862213 1255 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.863379 31449 consensus_queue.cc:237] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:42.864216 31451 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.865478 31319 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.866803 1604 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.867295 1255 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.868278 1255 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:42.870013 31187 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:42.871075 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5aa7bdac7bd1408cbf41d894d40aebd4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.871421 31320 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:42.871526 1604 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.872618 1604 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:42.874398 31319 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
W20240501 14:09:42.874806 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bca09920a7904362bc546916e3f6f5ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.881534 31188 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:42.886364 31451 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:42.889103 31044 catalog_manager.cc:5484] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:42.889415 31044 catalog_manager.cc:5484] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:42.897569 31319 consensus_queue.cc:237] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:42.899489 31451 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.899921 31580 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.900092 1663 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.900416 1555 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.904528 1555 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:42.905146 31451 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
W20240501 14:09:42.909155 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4603d1b1b1d441b1b0be0464e2f2c9e2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.909838 31046 catalog_manager.cc:5484] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:42.914661 31319 consensus_queue.cc:237] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:42.916214 31188 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.916220 31581 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.916615 1663 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.916982 970 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:42.918263 31256 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c8200d9c20e4140906c71c746badb94. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.919081 1663 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:42.919708 31581 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:42.922691 31046 catalog_manager.cc:5484] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:42.924690 31581 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:42.930243 31188 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:42.939666 31319 consensus_queue.cc:237] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:42.941488 31451 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.942516 31581 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.942610 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 14:09:42.943575 31256 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8cc4d3985224178aa008b4c9de5625b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.944077 1219 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.946514 1219 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.949028 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.949272 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.954128 31580 consensus_queue.cc:237] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:42.956451 31319 raft_consensus.cc:1261] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.956964 1589 consensus_queue.cc:1035] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.957288 31449 raft_consensus.cc:1261] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.957656 1589 consensus_queue.cc:1035] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.959287 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.962697 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 14:09:42.962864 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6bcd6a2f3c014d5ca0e32e16cdbc4edc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.962994 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.966575 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:42.979393 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:42.993747 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.999241 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.002748 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.003870 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:43.009436 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.022125 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:43.027303 1589 raft_consensus.cc:2932] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.030035 31449 raft_consensus.cc:2932] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.031095 31039 catalog_manager.cc:5484] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.039490 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.044088 31320 consensus_queue.cc:237] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:43.046962 31188 raft_consensus.cc:1261] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.047353 1219 consensus_queue.cc:1035] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.048735 31450 raft_consensus.cc:1261] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:43.049091 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b67bd0eb45a3486492b75ca3ad3be524. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.049423 1663 consensus_queue.cc:1035] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.053597 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:43.054443 31319 raft_consensus.cc:2932] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.060370 1663 raft_consensus.cc:2932] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.063017 31039 catalog_manager.cc:5484] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.065543 31188 raft_consensus.cc:2932] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.067445 1555 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.069226 31581 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.073063 31040 catalog_manager.cc:5484] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.073397 1760 ts_tablet_manager.cc:902] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:43.073465 31451 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.074148 1760 tablet_copy_client.cc:308] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:43.078739 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet 8f91909d3b1b46e9b858ce5c5b2e7975 from peer 9438789dc61f4501ad1d4ceaeccee458 ({username='slave'} at 127.23.191.70:41145)
I20240501 14:09:43.078904 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet 8f91909d3b1b46e9b858ce5c5b2e7975 from peer 9438789dc61f4501ad1d4ceaeccee458 at {username='slave'} at 127.23.191.70:41145: session id = 9438789dc61f4501ad1d4ceaeccee458-8f91909d3b1b46e9b858ce5c5b2e7975
I20240501 14:09:43.080286 31340 tablet_copy_source_session.cc:205] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.085584 31450 raft_consensus.cc:2932] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.093538 1760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f91909d3b1b46e9b858ce5c5b2e7975. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.100317 31188 consensus_queue.cc:237] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:43.101889 31450 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.102170 31319 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.102694 926 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.102869 1760 tablet_copy_client.cc:783] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.103052 1760 tablet_copy_client.cc:647] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.103283 926 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.105293 926 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
W20240501 14:09:43.106063 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0b7ede9ffaf496e9b106b5d05ed25a5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.108286 31043 catalog_manager.cc:5484] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.108420 1760 tablet_copy_client.cc:523] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.108549 31319 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.111032 31450 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.111821 1760 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap starting.
I20240501 14:09:43.120381 31319 consensus_queue.cc:237] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:43.120661 1760 log.cc:824] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:43.122061 31188 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.122486 1555 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.123837 31581 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.124990 1219 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.125633 970 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.127885 31188 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
W20240501 14:09:43.130167 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f26d16499ab54a3e9059f6e4ff041177. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.130503 31039 catalog_manager.cc:5484] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.136718 31188 consensus_queue.cc:237] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:43.138182 31189 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:43.138939 31581 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.141335 31581 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.141736 926 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.142432 31319 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.142828 926 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.145030 31580 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.145349 31320 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.145555 1767 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:43.145747 31124 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1006490ec2841b2a3365295caaf19e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.145756 926 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:43.146593 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7810938b8fb406e9c6ac438f54afd03. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.148715 1760 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:43.148962 1798 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.149129 1760 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap complete.
I20240501 14:09:43.149029 31581 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.149461 1760 ts_tablet_manager.cc:1367] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458: Time spent bootstrapping tablet: real 0.038s user 0.032s sys 0.004s
I20240501 14:09:43.151659 31043 catalog_manager.cc:5484] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.152231 1760 raft_consensus.cc:348] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.152503 1760 raft_consensus.cc:729] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Initialized, Role: LEARNER
I20240501 14:09:43.152873 1760 consensus_queue.cc:260] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.153259 926 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.153560 31581 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.153787 1760 ts_tablet_manager.cc:1398] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:09:43.155476 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session 9438789dc61f4501ad1d4ceaeccee458-8f91909d3b1b46e9b858ce5c5b2e7975 received from {username='slave'} at 127.23.191.70:41145
I20240501 14:09:43.155573 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session 9438789dc61f4501ad1d4ceaeccee458-8f91909d3b1b46e9b858ce5c5b2e7975 on tablet 8f91909d3b1b46e9b858ce5c5b2e7975 with peer 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:43.157624 31319 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.161273 31046 catalog_manager.cc:5484] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.166214 31319 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
W20240501 14:09:43.168733 1767 consensus_queue.cc:1159] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20240501 14:09:43.168941 31188 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
W20240501 14:09:43.170536 31124 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1006490ec2841b2a3365295caaf19e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.170687 31319 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.170907 31450 consensus_queue.cc:237] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:43.171039 1767 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:43.172443 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5aa7bdac7bd1408cbf41d894d40aebd4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.172506 31451 consensus_queue.cc:237] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
W20240501 14:09:43.173027 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5aa7bdac7bd1408cbf41d894d40aebd4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.172972 31319 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.174676 31581 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.175235 31189 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.175271 1604 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.175951 31046 catalog_manager.cc:5484] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 4 to 5, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.176736 31319 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.176779 31189 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.176887 1604 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.177320 1605 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.177592 1605 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:43.177652 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad0dac76fd0c4ae6b8e0549376c98e5d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.179286 31580 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.179301 926 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.179396 1605 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.179827 31319 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.180032 1255 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.181279 31580 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.182469 1817 ts_tablet_manager.cc:902] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:43.183665 31046 catalog_manager.cc:5484] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 4 to 5, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:43.184108 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1006490ec2841b2a3365295caaf19e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.184468 31187 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.185035 1817 tablet_copy_client.cc:308] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:43.186448 382 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.188169 31581 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.190593 31319 consensus_queue.cc:237] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
W20240501 14:09:43.192806 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8cc4d3985224178aa008b4c9de5625b. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:43.193439 31256 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8cc4d3985224178aa008b4c9de5625b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.193420 31039 catalog_manager.cc:5484] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.194437 31602 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet 9d2c173b2c32494eb16e46a51ca36e31 from peer b934fd09590a40ada094b9715d990a00 ({username='slave'} at 127.23.191.69:46931)
I20240501 14:09:43.194584 31602 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet 9d2c173b2c32494eb16e46a51ca36e31 from peer b934fd09590a40ada094b9715d990a00 at {username='slave'} at 127.23.191.69:46931: session id = b934fd09590a40ada094b9715d990a00-9d2c173b2c32494eb16e46a51ca36e31
I20240501 14:09:43.195989 31602 tablet_copy_source_session.cc:205] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.196848 31450 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.198781 970 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.199723 31580 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.200162 970 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.200613 31189 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.200662 1219 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.201104 31450 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.203696 1817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2c173b2c32494eb16e46a51ca36e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.209126 31043 catalog_manager.cc:5484] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 4 to 5, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.210281 31580 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.220453 1817 tablet_copy_client.cc:783] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.220692 1817 tablet_copy_client.cc:647] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.222426 31189 consensus_queue.cc:237] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.224653 31319 raft_consensus.cc:1261] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.225215 1798 consensus_queue.cc:1035] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.225677 31580 raft_consensus.cc:1261] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.226960 31189 consensus_queue.cc:237] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:43.227116 1798 consensus_queue.cc:1035] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.227964 31580 consensus_queue.cc:237] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.228485 31450 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.229279 926 raft_consensus.cc:2932] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.230712 1821 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.230794 31319 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:09:43.230994 31518 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d2c173b2c32494eb16e46a51ca36e31. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.231195 1214 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.231534 31450 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.231938 1214 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:43.232084 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.232375 31580 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.232793 31582 consensus_queue.cc:237] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:43.234388 1580 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.234584 31450 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.235399 31319 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.235512 31043 catalog_manager.cc:5484] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.237054 1589 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:43.237565 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28013d7df97943e09c27c3bb1794ee80. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:43.237679 31125 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b2e30cb9de14fe19273b3cef801cb67. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.238145 1798 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.238145 31319 raft_consensus.cc:2932] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.239037 31046 catalog_manager.cc:5484] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.239349 31582 raft_consensus.cc:2932] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
W20240501 14:09:43.240147 31518 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ad9c714b2b4fff94063d2838835173. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.240398 31450 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.241075 31189 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.241461 1214 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.243016 1817 tablet_copy_client.cc:523] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.246381 1214 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.247396 1817 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00: Bootstrap starting.
I20240501 14:09:43.248488 31187 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.250592 31319 consensus_queue.cc:237] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.251838 31040 catalog_manager.cc:5484] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
W20240501 14:09:43.252470 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0399a419a25c4981a2ee90d76cbf59b1. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:43.252825 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0399a419a25c4981a2ee90d76cbf59b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.253095 31187 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.253469 1663 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.254185 31450 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.254669 1663 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.255048 1219 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.255584 31449 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.257046 31451 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.257972 1798 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.258875 31187 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.258971 31039 catalog_manager.cc:5484] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.260187 31580 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.263303 31451 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.264067 31046 catalog_manager.cc:5484] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.265671 1817 log.cc:824] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:43.274215 31187 consensus_queue.cc:237] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.276353 31319 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.277032 1255 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.277119 31580 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:09:43.277410 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7810938b8fb406e9c6ac438f54afd03. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.277591 1255 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:43.277887 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7810938b8fb406e9c6ac438f54afd03. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.278954 1255 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.279779 31580 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.282232 31046 catalog_manager.cc:5484] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.283257 31189 consensus_queue.cc:237] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:43.285316 31451 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:09:43.285382 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bca09920a7904362bc546916e3f6f5ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.285837 31189 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.286121 31320 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:09:43.286535 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bca09920a7904362bc546916e3f6f5ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.286794 926 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.286789 31319 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.287626 1798 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:43.287763 31124 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1006490ec2841b2a3365295caaf19e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.288226 31580 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:43.289076 1767 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:43.289239 926 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
W20240501 14:09:43.289469 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1006490ec2841b2a3365295caaf19e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.289486 31320 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:43.289791 31451 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
W20240501 14:09:43.290375 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1006490ec2841b2a3365295caaf19e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.290493 1767 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:43.293502 31039 catalog_manager.cc:5484] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 4 to 5, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.293820 1767 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.295802 31319 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.298561 31319 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.298631 31580 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.299644 31046 catalog_manager.cc:5484] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 5 to 6, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.301350 1901 ts_tablet_manager.cc:902] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:43.302829 1901 tablet_copy_client.cc:308] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:39413
I20240501 14:09:43.307299 31189 consensus_queue.cc:237] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:43.312207 31580 raft_consensus.cc:1261] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:43.312263 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71241540f585474a8f62da16ed2d9874. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.314113 31319 raft_consensus.cc:1261] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.315181 926 consensus_queue.cc:1035] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.315537 926 consensus_queue.cc:1035] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.317579 926 raft_consensus.cc:2932] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.319726 31320 raft_consensus.cc:2932] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.320345 31039 catalog_manager.cc:5484] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.324571 31320 consensus_queue.cc:237] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:43.326241 31582 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.327004 31189 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.327407 1555 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.328437 970 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:43.329041 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d994081c3414f779ba2cae431cb1d97. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.331225 970 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.332218 31582 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.333425 31581 consensus_queue.cc:237] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
W20240501 14:09:43.335431 31518 consensus_peers.cc:485] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c082ccc777ba4f9a937931f707645e54. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.335805 31319 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.336232 31471 tablet_copy_service.cc:140] P 6a6fcc099d1340c383e5a429643d5b7c: Received BeginTabletCopySession request for tablet 7a542c07a3f841b0a31051a8a37083bf from peer a63e0dd1ba2e4a8db8ace9718a700c32 ({username='slave'} at 127.23.191.72:40223)
I20240501 14:09:43.336238 31046 catalog_manager.cc:5484] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.336576 31451 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.336704 31471 tablet_copy_service.cc:161] P 6a6fcc099d1340c383e5a429643d5b7c: Beginning new tablet copy session on tablet 7a542c07a3f841b0a31051a8a37083bf from peer a63e0dd1ba2e4a8db8ace9718a700c32 at {username='slave'} at 127.23.191.72:40223: session id = a63e0dd1ba2e4a8db8ace9718a700c32-7a542c07a3f841b0a31051a8a37083bf
I20240501 14:09:43.337801 1580 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.338218 31471 tablet_copy_source_session.cc:205] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.338932 1580 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.342875 31319 consensus_queue.cc:237] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
W20240501 14:09:43.344455 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d994081c3414f779ba2cae431cb1d97. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.344460 31581 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.343926 1901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a542c07a3f841b0a31051a8a37083bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:09:43.344870 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d994081c3414f779ba2cae431cb1d97. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.344976 1663 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.345444 31187 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.346195 1663 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.346948 1663 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.346951 31187 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.349423 31581 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.349687 31043 catalog_manager.cc:5484] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.352274 1817 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:43.352725 1817 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00: Bootstrap complete.
I20240501 14:09:43.353044 1817 ts_tablet_manager.cc:1367] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00: Time spent bootstrapping tablet: real 0.106s user 0.037s sys 0.000s
I20240501 14:09:43.354952 31187 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.355448 1817 raft_consensus.cc:348] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:43.355675 1817 raft_consensus.cc:729] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b934fd09590a40ada094b9715d990a00, State: Initialized, Role: LEARNER
I20240501 14:09:43.355996 1817 consensus_queue.cc:260] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:43.361069 1582 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.364065 31451 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.365844 1817 ts_tablet_manager.cc:1398] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240501 14:09:43.366519 31320 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.367748 31039 catalog_manager.cc:5484] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.368302 1760 ts_tablet_manager.cc:902] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:43.368737 1760 tablet_copy_client.cc:308] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:43.369582 31580 raft_consensus.cc:2932] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:43.372661 31602 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session b934fd09590a40ada094b9715d990a00-9d2c173b2c32494eb16e46a51ca36e31 received from {username='slave'} at 127.23.191.69:46931
I20240501 14:09:43.372749 31602 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session b934fd09590a40ada094b9715d990a00-9d2c173b2c32494eb16e46a51ca36e31 on tablet 9d2c173b2c32494eb16e46a51ca36e31 with peer b934fd09590a40ada094b9715d990a00
I20240501 14:09:43.374311 1817 ts_tablet_manager.cc:902] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:43.374730 1817 tablet_copy_client.cc:308] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:43.375676 31602 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1 from peer b934fd09590a40ada094b9715d990a00 ({username='slave'} at 127.23.191.69:46931)
I20240501 14:09:43.375766 31602 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1 from peer b934fd09590a40ada094b9715d990a00 at {username='slave'} at 127.23.191.69:46931: session id = b934fd09590a40ada094b9715d990a00-b83f4cd48b4b4251a8ac4ddebdb8e4e1
I20240501 14:09:43.375870 31601 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet 6bcd6a2f3c014d5ca0e32e16cdbc4edc from peer 9438789dc61f4501ad1d4ceaeccee458 ({username='slave'} at 127.23.191.70:37219)
I20240501 14:09:43.375932 31601 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet 6bcd6a2f3c014d5ca0e32e16cdbc4edc from peer 9438789dc61f4501ad1d4ceaeccee458 at {username='slave'} at 127.23.191.70:37219: session id = 9438789dc61f4501ad1d4ceaeccee458-6bcd6a2f3c014d5ca0e32e16cdbc4edc
I20240501 14:09:43.376616 31602 tablet_copy_source_session.cc:205] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.376662 31601 tablet_copy_source_session.cc:205] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.377473 1760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bcd6a2f3c014d5ca0e32e16cdbc4edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.377668 1817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.379556 1760 tablet_copy_client.cc:783] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.379551 31187 consensus_queue.cc:237] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:43.379695 1760 tablet_copy_client.cc:647] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.380347 1817 tablet_copy_client.cc:783] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.380478 1817 tablet_copy_client.cc:647] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.381088 31580 consensus_queue.cc:237] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
W20240501 14:09:43.381237 31124 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b2e30cb9de14fe19273b3cef801cb67. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.381597 31581 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.381939 1767 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:43.382262 31125 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b2e30cb9de14fe19273b3cef801cb67. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.382493 31451 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.382853 31449 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.383009 1767 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.383270 1214 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.383837 1901 tablet_copy_client.cc:783] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.384021 1901 tablet_copy_client.cc:647] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.384068 31320 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:43.384811 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e68be1521964ba8b864801fdf3b0df3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.384707 1895 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.385051 1214 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.385612 31449 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.388368 31043 catalog_manager.cc:5484] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.388443 1214 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.390197 31580 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.390581 1817 tablet_copy_client.cc:523] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.391053 1760 tablet_copy_client.cc:523] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.392395 1760 tablet_bootstrap.cc:492] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap starting.
I20240501 14:09:43.393893 1817 tablet_bootstrap.cc:492] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00: Bootstrap starting.
I20240501 14:09:43.397058 31449 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.397389 31320 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:43.399466 1901 tablet_copy_client.cc:523] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.399037 31043 catalog_manager.cc:5484] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.401562 1901 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap starting.
I20240501 14:09:43.415547 31580 consensus_queue.cc:237] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
W20240501 14:09:43.417393 31518 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ad9c714b2b4fff94063d2838835173. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.417924 31449 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.418339 1589 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.420310 1589 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.421216 31187 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.421772 1582 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:43.421929 31517 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ad9c714b2b4fff94063d2838835173. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.422623 31187 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.424593 31039 catalog_manager.cc:5484] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.425161 1817 tablet_bootstrap.cc:492] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:43.425417 1817 tablet_bootstrap.cc:492] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00: Bootstrap complete.
I20240501 14:09:43.425575 1817 ts_tablet_manager.cc:1367] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.000s
I20240501 14:09:43.426121 1817 raft_consensus.cc:348] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:43.426229 1817 raft_consensus.cc:729] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b934fd09590a40ada094b9715d990a00, State: Initialized, Role: LEARNER
I20240501 14:09:43.426334 1817 consensus_queue.cc:260] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:43.426602 1817 ts_tablet_manager.cc:1398] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:43.427011 31601 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session b934fd09590a40ada094b9715d990a00-b83f4cd48b4b4251a8ac4ddebdb8e4e1 received from {username='slave'} at 127.23.191.69:46931
I20240501 14:09:43.427083 31601 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session b934fd09590a40ada094b9715d990a00-b83f4cd48b4b4251a8ac4ddebdb8e4e1 on tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1 with peer b934fd09590a40ada094b9715d990a00
I20240501 14:09:43.428923 31449 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.440631 31187 consensus_queue.cc:237] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:43.442200 31449 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.443188 31580 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:43.443583 1255 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:43.444104 31124 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96f5e4e25a674a99a6762757f0ab1545. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.445230 1255 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:43.454351 2002 ts_tablet_manager.cc:902] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:43.454638 1901 log.cc:824] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:43.455065 1985 ts_tablet_manager.cc:902] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:43.455971 1985 tablet_copy_client.cc:308] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:43.455564 2002 tablet_copy_client.cc:308] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:43.458010 1767 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.458354 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet 3c8200d9c20e4140906c71c746badb94 from peer 9438789dc61f4501ad1d4ceaeccee458 ({username='slave'} at 127.23.191.70:41145)
I20240501 14:09:43.458459 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet 3c8200d9c20e4140906c71c746badb94 from peer 9438789dc61f4501ad1d4ceaeccee458 at {username='slave'} at 127.23.191.70:41145: session id = 9438789dc61f4501ad1d4ceaeccee458-3c8200d9c20e4140906c71c746badb94
I20240501 14:09:43.459445 31340 tablet_copy_source_session.cc:205] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.460408 2002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8200d9c20e4140906c71c746badb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.460355 31580 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.462512 31451 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.462767 31040 catalog_manager.cc:5484] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:43.466347 1760 tablet_bootstrap.cc:492] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:43.466586 1760 tablet_bootstrap.cc:492] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap complete.
I20240501 14:09:43.466735 1760 ts_tablet_manager.cc:1367] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458: Time spent bootstrapping tablet: real 0.074s user 0.020s sys 0.004s
I20240501 14:09:43.467329 1760 raft_consensus.cc:348] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.467458 1760 raft_consensus.cc:729] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Initialized, Role: LEARNER
I20240501 14:09:43.467643 2002 tablet_copy_client.cc:783] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.467839 2002 tablet_copy_client.cc:647] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.469808 1760 consensus_queue.cc:260] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.470544 1760 ts_tablet_manager.cc:1398] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:43.471035 31601 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session 9438789dc61f4501ad1d4ceaeccee458-6bcd6a2f3c014d5ca0e32e16cdbc4edc received from {username='slave'} at 127.23.191.70:37219
I20240501 14:09:43.471123 31601 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session 9438789dc61f4501ad1d4ceaeccee458-6bcd6a2f3c014d5ca0e32e16cdbc4edc on tablet 6bcd6a2f3c014d5ca0e32e16cdbc4edc with peer 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:43.475880 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:59967)
I20240501 14:09:43.475998 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:59967: session id = a0caf2974041408bb32362bfc4284eaa-4603d1b1b1d441b1b0be0464e2f2c9e2
I20240501 14:09:43.476899 31340 tablet_copy_source_session.cc:205] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.480379 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.492399 2002 tablet_copy_client.cc:523] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.494300 2002 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap starting.
I20240501 14:09:43.495090 31187 consensus_queue.cc:237] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.496999 31449 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.498147 1895 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.499131 31320 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:09:43.499544 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0b7ede9ffaf496e9b106b5d05ed25a5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.499555 1895 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240501 14:09:43.504813 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0b7ede9ffaf496e9b106b5d05ed25a5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.506631 1895 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.508726 31449 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.509536 31039 catalog_manager.cc:5484] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.510321 1985 tablet_copy_client.cc:783] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.510947 31320 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.518307 1817 ts_tablet_manager.cc:902] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:43.518702 1817 tablet_copy_client.cc:308] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:43.519716 1985 tablet_copy_client.cc:647] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.520910 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:43.525012 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet f26d16499ab54a3e9059f6e4ff041177 from peer b934fd09590a40ada094b9715d990a00 ({username='slave'} at 127.23.191.69:54165)
I20240501 14:09:43.525113 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet f26d16499ab54a3e9059f6e4ff041177 from peer b934fd09590a40ada094b9715d990a00 at {username='slave'} at 127.23.191.69:54165: session id = b934fd09590a40ada094b9715d990a00-f26d16499ab54a3e9059f6e4ff041177
I20240501 14:09:43.525949 31340 tablet_copy_source_session.cc:205] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.526721 1985 tablet_copy_client.cc:523] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.527840 1817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26d16499ab54a3e9059f6e4ff041177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.529940 1817 tablet_copy_client.cc:783] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.530084 1817 tablet_copy_client.cc:647] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.531644 1985 tablet_bootstrap.cc:492] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:43.533226 1817 tablet_copy_client.cc:523] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.533830 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.534358 1817 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00: Bootstrap starting.
I20240501 14:09:43.549774 1985 log.cc:824] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:43.553200 2002 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:43.553526 2002 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap complete.
I20240501 14:09:43.553726 2002 ts_tablet_manager.cc:1367] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458: Time spent bootstrapping tablet: real 0.059s user 0.027s sys 0.000s
I20240501 14:09:43.554373 2002 raft_consensus.cc:348] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.554556 2002 raft_consensus.cc:729] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Initialized, Role: LEARNER
I20240501 14:09:43.554689 2002 consensus_queue.cc:260] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.555521 2002 ts_tablet_manager.cc:1398] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:43.556193 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session 9438789dc61f4501ad1d4ceaeccee458-3c8200d9c20e4140906c71c746badb94 received from {username='slave'} at 127.23.191.70:41145
I20240501 14:09:43.556281 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session 9438789dc61f4501ad1d4ceaeccee458-3c8200d9c20e4140906c71c746badb94 on tablet 3c8200d9c20e4140906c71c746badb94 with peer 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:43.559188 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:43.559995 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.561501 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.562484 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.606657 475 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:43.608085 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:43.640653 2078 ts_tablet_manager.cc:902] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:43.641265 2078 tablet_copy_client.cc:308] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:39413
I20240501 14:09:43.641755 31471 tablet_copy_service.cc:140] P 6a6fcc099d1340c383e5a429643d5b7c: Received BeginTabletCopySession request for tablet 5aa7bdac7bd1408cbf41d894d40aebd4 from peer a63e0dd1ba2e4a8db8ace9718a700c32 ({username='slave'} at 127.23.191.72:40223)
I20240501 14:09:43.641857 31471 tablet_copy_service.cc:161] P 6a6fcc099d1340c383e5a429643d5b7c: Beginning new tablet copy session on tablet 5aa7bdac7bd1408cbf41d894d40aebd4 from peer a63e0dd1ba2e4a8db8ace9718a700c32 at {username='slave'} at 127.23.191.72:40223: session id = a63e0dd1ba2e4a8db8ace9718a700c32-5aa7bdac7bd1408cbf41d894d40aebd4
I20240501 14:09:43.642822 31471 tablet_copy_source_session.cc:205] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.642897 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:43.645854 2078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa7bdac7bd1408cbf41d894d40aebd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.649772 2078 tablet_copy_client.cc:783] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.649978 2078 tablet_copy_client.cc:647] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.651146 2078 tablet_copy_client.cc:523] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.652585 2078 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap starting.
I20240501 14:09:43.652755 606 raft_consensus.cc:1203] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:43.654587 1901 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:43.656680 2002 ts_tablet_manager.cc:902] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:43.657091 2002 tablet_copy_client.cc:308] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:43.657897 31601 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet 9d2c173b2c32494eb16e46a51ca36e31 from peer 9438789dc61f4501ad1d4ceaeccee458 ({username='slave'} at 127.23.191.70:37219)
I20240501 14:09:43.658025 31601 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet 9d2c173b2c32494eb16e46a51ca36e31 from peer 9438789dc61f4501ad1d4ceaeccee458 at {username='slave'} at 127.23.191.70:37219: session id = 9438789dc61f4501ad1d4ceaeccee458-9d2c173b2c32494eb16e46a51ca36e31
I20240501 14:09:43.658978 31601 tablet_copy_source_session.cc:205] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.659874 2002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2c173b2c32494eb16e46a51ca36e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.659919 1901 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap complete.
I20240501 14:09:43.660236 1901 ts_tablet_manager.cc:1367] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent bootstrapping tablet: real 0.259s user 0.038s sys 0.000s
I20240501 14:09:43.661911 2002 tablet_copy_client.cc:783] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.662032 2002 tablet_copy_client.cc:647] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.663290 1901 raft_consensus.cc:348] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:43.665022 1901 raft_consensus.cc:729] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63e0dd1ba2e4a8db8ace9718a700c32, State: Initialized, Role: LEARNER
I20240501 14:09:43.666100 1901 consensus_queue.cc:260] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:43.667191 1901 ts_tablet_manager.cc:1398] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:09:43.667977 31471 tablet_copy_service.cc:342] P 6a6fcc099d1340c383e5a429643d5b7c: Request end of tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-7a542c07a3f841b0a31051a8a37083bf received from {username='slave'} at 127.23.191.72:40223
I20240501 14:09:43.668066 31471 tablet_copy_service.cc:433] P 6a6fcc099d1340c383e5a429643d5b7c: ending tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-7a542c07a3f841b0a31051a8a37083bf on tablet 7a542c07a3f841b0a31051a8a37083bf with peer a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:43.669095 1760 ts_tablet_manager.cc:902] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:43.669598 1760 tablet_copy_client.cc:308] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:43.671515 2081 ts_tablet_manager.cc:902] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:43.672124 2081 tablet_copy_client.cc:308] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:43.672711 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet b67bd0eb45a3486492b75ca3ad3be524 from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:59967)
I20240501 14:09:43.672820 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet b67bd0eb45a3486492b75ca3ad3be524 from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:59967: session id = a0caf2974041408bb32362bfc4284eaa-b67bd0eb45a3486492b75ca3ad3be524
I20240501 14:09:43.673741 31340 tablet_copy_source_session.cc:205] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.675101 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet 28013d7df97943e09c27c3bb1794ee80 from peer 9438789dc61f4501ad1d4ceaeccee458 ({username='slave'} at 127.23.191.70:36823)
I20240501 14:09:43.675266 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet 28013d7df97943e09c27c3bb1794ee80 from peer 9438789dc61f4501ad1d4ceaeccee458 at {username='slave'} at 127.23.191.70:36823: session id = 9438789dc61f4501ad1d4ceaeccee458-28013d7df97943e09c27c3bb1794ee80
I20240501 14:09:43.676673 31209 tablet_copy_source_session.cc:205] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.682310 1760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28013d7df97943e09c27c3bb1794ee80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.684576 1760 tablet_copy_client.cc:783] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.684717 1760 tablet_copy_client.cc:647] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.685173 2002 tablet_copy_client.cc:523] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.686421 2002 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap starting.
I20240501 14:09:43.689297 1927 ts_tablet_manager.cc:902] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:43.689872 1927 tablet_copy_client.cc:308] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:43.697345 2122 ts_tablet_manager.cc:902] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:43.699455 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet a7810938b8fb406e9c6ac438f54afd03 from peer b934fd09590a40ada094b9715d990a00 ({username='slave'} at 127.23.191.69:37295)
I20240501 14:09:43.699589 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet a7810938b8fb406e9c6ac438f54afd03 from peer b934fd09590a40ada094b9715d990a00 at {username='slave'} at 127.23.191.69:37295: session id = b934fd09590a40ada094b9715d990a00-a7810938b8fb406e9c6ac438f54afd03
I20240501 14:09:43.700153 2122 tablet_copy_client.cc:308] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:43.700577 31209 tablet_copy_source_session.cc:205] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.700667 31208 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet a7810938b8fb406e9c6ac438f54afd03 from peer 9438789dc61f4501ad1d4ceaeccee458 ({username='slave'} at 127.23.191.70:36823)
I20240501 14:09:43.700773 31208 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet a7810938b8fb406e9c6ac438f54afd03 from peer 9438789dc61f4501ad1d4ceaeccee458 at {username='slave'} at 127.23.191.70:36823: session id = 9438789dc61f4501ad1d4ceaeccee458-a7810938b8fb406e9c6ac438f54afd03
I20240501 14:09:43.701689 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7810938b8fb406e9c6ac438f54afd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.701700 31208 tablet_copy_source_session.cc:205] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.704286 1927 tablet_copy_client.cc:783] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.704344 2122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7810938b8fb406e9c6ac438f54afd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.704479 1927 tablet_copy_client.cc:647] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.706663 2122 tablet_copy_client.cc:783] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.706867 2122 tablet_copy_client.cc:647] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.708427 2122 tablet_copy_client.cc:523] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.709851 2122 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap starting.
I20240501 14:09:43.722863 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:43.726929 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:43.738135 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.741042 2143 ts_tablet_manager.cc:902] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:43.745056 2134 ts_tablet_manager.cc:902] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:43.745716 2134 tablet_copy_client.cc:308] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:39413
I20240501 14:09:43.745779 2142 ts_tablet_manager.cc:902] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:43.746023 1760 tablet_copy_client.cc:523] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.746348 2142 tablet_copy_client.cc:308] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:43.746800 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet f8cc4d3985224178aa008b4c9de5625b from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:59967)
I20240501 14:09:43.746904 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet f8cc4d3985224178aa008b4c9de5625b from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:59967: session id = a0caf2974041408bb32362bfc4284eaa-f8cc4d3985224178aa008b4c9de5625b
I20240501 14:09:43.747782 1760 tablet_bootstrap.cc:492] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap starting.
I20240501 14:09:43.747861 31340 tablet_copy_source_session.cc:205] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.748818 2142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cc4d3985224178aa008b4c9de5625b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.750156 1901 ts_tablet_manager.cc:902] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:43.750545 1901 tablet_copy_client.cc:308] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:43.751430 2142 tablet_copy_client.cc:783] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.751647 2142 tablet_copy_client.cc:647] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.751974 2152 ts_tablet_manager.cc:902] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:43.752540 2152 tablet_copy_client.cc:308] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:39413
I20240501 14:09:43.756664 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet 0399a419a25c4981a2ee90d76cbf59b1 from peer a63e0dd1ba2e4a8db8ace9718a700c32 ({username='slave'} at 127.23.191.72:44631)
I20240501 14:09:43.756783 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet 0399a419a25c4981a2ee90d76cbf59b1 from peer a63e0dd1ba2e4a8db8ace9718a700c32 at {username='slave'} at 127.23.191.72:44631: session id = a63e0dd1ba2e4a8db8ace9718a700c32-0399a419a25c4981a2ee90d76cbf59b1
I20240501 14:09:43.757709 31340 tablet_copy_source_session.cc:205] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.758011 2081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b67bd0eb45a3486492b75ca3ad3be524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.758867 1901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0399a419a25c4981a2ee90d76cbf59b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.760354 2081 tablet_copy_client.cc:783] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.760566 2081 tablet_copy_client.cc:647] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.761000 1901 tablet_copy_client.cc:783] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.761142 1901 tablet_copy_client.cc:647] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.761399 31471 tablet_copy_service.cc:140] P 6a6fcc099d1340c383e5a429643d5b7c: Received BeginTabletCopySession request for tablet 5aa7bdac7bd1408cbf41d894d40aebd4 from peer b934fd09590a40ada094b9715d990a00 ({username='slave'} at 127.23.191.69:47269)
I20240501 14:09:43.761487 31471 tablet_copy_service.cc:161] P 6a6fcc099d1340c383e5a429643d5b7c: Beginning new tablet copy session on tablet 5aa7bdac7bd1408cbf41d894d40aebd4 from peer b934fd09590a40ada094b9715d990a00 at {username='slave'} at 127.23.191.69:47269: session id = b934fd09590a40ada094b9715d990a00-5aa7bdac7bd1408cbf41d894d40aebd4
I20240501 14:09:43.762388 31471 tablet_copy_source_session.cc:205] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.763496 31471 tablet_copy_service.cc:140] P 6a6fcc099d1340c383e5a429643d5b7c: Received BeginTabletCopySession request for tablet ad0dac76fd0c4ae6b8e0549376c98e5d from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:41729)
I20240501 14:09:43.763585 31471 tablet_copy_service.cc:161] P 6a6fcc099d1340c383e5a429643d5b7c: Beginning new tablet copy session on tablet ad0dac76fd0c4ae6b8e0549376c98e5d from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:41729: session id = a0caf2974041408bb32362bfc4284eaa-ad0dac76fd0c4ae6b8e0549376c98e5d
I20240501 14:09:43.764381 31471 tablet_copy_source_session.cc:205] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.764423 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa7bdac7bd1408cbf41d894d40aebd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.764652 2142 tablet_copy_client.cc:523] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.765478 1901 tablet_copy_client.cc:523] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.766695 1901 tablet_bootstrap.cc:492] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap starting.
I20240501 14:09:43.767675 2134 tablet_copy_client.cc:783] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.767817 2134 tablet_copy_client.cc:647] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.769950 2142 tablet_bootstrap.cc:492] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:43.771467 475 raft_consensus.cc:1203] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:43.775995 2152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad0dac76fd0c4ae6b8e0549376c98e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.776361 606 raft_consensus.cc:1203] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:43.777611 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.781446 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.785007 2144 ts_tablet_manager.cc:902] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:43.785529 2144 tablet_copy_client.cc:308] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:43.786139 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet b1006490ec2841b2a3365295caaf19e1 from peer b934fd09590a40ada094b9715d990a00 ({username='slave'} at 127.23.191.69:37295)
I20240501 14:09:43.786237 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet b1006490ec2841b2a3365295caaf19e1 from peer b934fd09590a40ada094b9715d990a00 at {username='slave'} at 127.23.191.69:37295: session id = b934fd09590a40ada094b9715d990a00-b1006490ec2841b2a3365295caaf19e1
I20240501 14:09:43.787119 31209 tablet_copy_source_session.cc:205] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.789798 2152 tablet_copy_client.cc:783] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.790017 2152 tablet_copy_client.cc:647] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.790768 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:43.794157 868 raft_consensus.cc:1203] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:43.799099 2152 tablet_copy_client.cc:523] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.800520 2152 tablet_bootstrap.cc:492] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:43.810618 2002 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:43.810657 2134 tablet_copy_client.cc:523] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.812076 2134 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00: Bootstrap starting.
I20240501 14:09:43.815764 2002 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap complete.
I20240501 14:09:43.815946 2002 ts_tablet_manager.cc:1367] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458: Time spent bootstrapping tablet: real 0.130s user 0.028s sys 0.000s
I20240501 14:09:43.816416 2002 raft_consensus.cc:348] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.816558 2002 raft_consensus.cc:729] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Initialized, Role: LEARNER
I20240501 14:09:43.816671 2002 consensus_queue.cc:260] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.816988 2002 ts_tablet_manager.cc:1398] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.819798 31601 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session 9438789dc61f4501ad1d4ceaeccee458-9d2c173b2c32494eb16e46a51ca36e31 received from {username='slave'} at 127.23.191.70:37219
I20240501 14:09:43.819886 31601 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session 9438789dc61f4501ad1d4ceaeccee458-9d2c173b2c32494eb16e46a51ca36e31 on tablet 9d2c173b2c32494eb16e46a51ca36e31 with peer 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:43.821046 2144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1006490ec2841b2a3365295caaf19e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.824710 2144 tablet_copy_client.cc:783] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.824952 2144 tablet_copy_client.cc:647] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.828473 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:43.828483 2143 tablet_copy_client.cc:308] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:43.812481 2122 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:43.829252 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet 0399a419a25c4981a2ee90d76cbf59b1 from peer 9438789dc61f4501ad1d4ceaeccee458 ({username='slave'} at 127.23.191.70:41145)
I20240501 14:09:43.829371 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet 0399a419a25c4981a2ee90d76cbf59b1 from peer 9438789dc61f4501ad1d4ceaeccee458 at {username='slave'} at 127.23.191.70:41145: session id = 9438789dc61f4501ad1d4ceaeccee458-0399a419a25c4981a2ee90d76cbf59b1
I20240501 14:09:43.829486 2122 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap complete.
I20240501 14:09:43.829738 2122 ts_tablet_manager.cc:1367] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458: Time spent bootstrapping tablet: real 0.120s user 0.027s sys 0.000s
I20240501 14:09:43.830274 31340 tablet_copy_source_session.cc:205] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.830983 2002 ts_tablet_manager.cc:902] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:43.831385 2002 tablet_copy_client.cc:308] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:43.831499 2122 raft_consensus.cc:348] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.831820 2122 raft_consensus.cc:729] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Initialized, Role: LEARNER
I20240501 14:09:43.831856 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet f8cc4d3985224178aa008b4c9de5625b from peer 9438789dc61f4501ad1d4ceaeccee458 ({username='slave'} at 127.23.191.70:41145)
I20240501 14:09:43.831949 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet f8cc4d3985224178aa008b4c9de5625b from peer 9438789dc61f4501ad1d4ceaeccee458 at {username='slave'} at 127.23.191.70:41145: session id = 9438789dc61f4501ad1d4ceaeccee458-f8cc4d3985224178aa008b4c9de5625b
I20240501 14:09:43.832698 31340 tablet_copy_source_session.cc:205] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.832036 2122 consensus_queue.cc:260] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.833895 2122 ts_tablet_manager.cc:1398] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:43.835873 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session 9438789dc61f4501ad1d4ceaeccee458-a7810938b8fb406e9c6ac438f54afd03 received from {username='slave'} at 127.23.191.70:36823
I20240501 14:09:43.835965 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session 9438789dc61f4501ad1d4ceaeccee458-a7810938b8fb406e9c6ac438f54afd03 on tablet a7810938b8fb406e9c6ac438f54afd03 with peer 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:43.837080 2002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cc4d3985224178aa008b4c9de5625b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.840431 2002 tablet_copy_client.cc:783] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.840622 2002 tablet_copy_client.cc:647] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.841684 2195 ts_tablet_manager.cc:902] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:43.842307 2195 tablet_copy_client.cc:308] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:43.847016 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet 71241540f585474a8f62da16ed2d9874 from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:42437)
I20240501 14:09:43.847126 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet 71241540f585474a8f62da16ed2d9874 from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:42437: session id = a0caf2974041408bb32362bfc4284eaa-71241540f585474a8f62da16ed2d9874
I20240501 14:09:43.848080 31209 tablet_copy_source_session.cc:205] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.848938 2122 ts_tablet_manager.cc:902] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:43.849673 2195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71241540f585474a8f62da16ed2d9874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.852172 2195 tablet_copy_client.cc:783] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.852378 2195 tablet_copy_client.cc:647] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.852481 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.853629 2195 tablet_copy_client.cc:523] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.854271 2122 tablet_copy_client.cc:308] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:43.854948 2195 tablet_bootstrap.cc:492] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:43.869906 1901 tablet_bootstrap.cc:492] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:43.871636 2144 tablet_copy_client.cc:523] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.874395 2144 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00: Bootstrap starting.
I20240501 14:09:43.874577 1901 tablet_bootstrap.cc:492] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap complete.
I20240501 14:09:43.875011 1901 ts_tablet_manager.cc:1367] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent bootstrapping tablet: real 0.108s user 0.026s sys 0.000s
I20240501 14:09:43.875778 1901 raft_consensus.cc:348] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.876009 1901 raft_consensus.cc:729] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63e0dd1ba2e4a8db8ace9718a700c32, State: Initialized, Role: LEARNER
I20240501 14:09:43.876161 1901 consensus_queue.cc:260] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:43.877317 1901 ts_tablet_manager.cc:1398] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:43.877866 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-0399a419a25c4981a2ee90d76cbf59b1 received from {username='slave'} at 127.23.191.72:44631
I20240501 14:09:43.877943 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-0399a419a25c4981a2ee90d76cbf59b1 on tablet 0399a419a25c4981a2ee90d76cbf59b1 with peer a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:43.878233 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet b1006490ec2841b2a3365295caaf19e1 from peer 9438789dc61f4501ad1d4ceaeccee458 ({username='slave'} at 127.23.191.70:36823)
I20240501 14:09:43.878332 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet b1006490ec2841b2a3365295caaf19e1 from peer 9438789dc61f4501ad1d4ceaeccee458 at {username='slave'} at 127.23.191.70:36823: session id = 9438789dc61f4501ad1d4ceaeccee458-b1006490ec2841b2a3365295caaf19e1
I20240501 14:09:43.878477 2143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0399a419a25c4981a2ee90d76cbf59b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.879160 31209 tablet_copy_source_session.cc:205] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.881774 606 raft_consensus.cc:1203] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:43.882882 1901 ts_tablet_manager.cc:902] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:43.884107 2143 tablet_copy_client.cc:783] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.884305 2143 tablet_copy_client.cc:647] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.889595 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:43.896971 2143 tablet_copy_client.cc:523] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.902750 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.906141 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:43.907778 1901 tablet_copy_client.cc:308] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:43.909590 2122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1006490ec2841b2a3365295caaf19e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.911746 2122 tablet_copy_client.cc:783] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.911885 2122 tablet_copy_client.cc:647] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.912714 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet bca09920a7904362bc546916e3f6f5ac from peer a63e0dd1ba2e4a8db8ace9718a700c32 ({username='slave'} at 127.23.191.72:59731)
I20240501 14:09:43.912814 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet bca09920a7904362bc546916e3f6f5ac from peer a63e0dd1ba2e4a8db8ace9718a700c32 at {username='slave'} at 127.23.191.72:59731: session id = a63e0dd1ba2e4a8db8ace9718a700c32-bca09920a7904362bc546916e3f6f5ac
I20240501 14:09:43.913360 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.913749 31209 tablet_copy_source_session.cc:205] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.915026 1901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca09920a7904362bc546916e3f6f5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.917940 2122 tablet_copy_client.cc:523] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.919104 2122 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap starting.
I20240501 14:09:43.927968 31299 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:43.935432 2194 ts_tablet_manager.cc:902] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:43.936203 2194 tablet_copy_client.cc:308] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:43.937350 2189 ts_tablet_manager.cc:902] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:43.937625 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.937968 2189 tablet_copy_client.cc:308] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:43.938424 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet 5b2e30cb9de14fe19273b3cef801cb67 from peer b934fd09590a40ada094b9715d990a00 ({username='slave'} at 127.23.191.69:37295)
I20240501 14:09:43.938526 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet 5b2e30cb9de14fe19273b3cef801cb67 from peer b934fd09590a40ada094b9715d990a00 at {username='slave'} at 127.23.191.69:37295: session id = b934fd09590a40ada094b9715d990a00-5b2e30cb9de14fe19273b3cef801cb67
I20240501 14:09:43.939450 31209 tablet_copy_source_session.cc:205] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.940341 2189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2e30cb9de14fe19273b3cef801cb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.942766 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:43.949394 1927 tablet_copy_client.cc:523] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.950706 1927 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00: Bootstrap starting.
I20240501 14:09:43.952963 31601 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet c082ccc777ba4f9a937931f707645e54 from peer a63e0dd1ba2e4a8db8ace9718a700c32 ({username='slave'} at 127.23.191.72:34905)
I20240501 14:09:43.953063 31601 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet c082ccc777ba4f9a937931f707645e54 from peer a63e0dd1ba2e4a8db8ace9718a700c32 at {username='slave'} at 127.23.191.72:34905: session id = a63e0dd1ba2e4a8db8ace9718a700c32-c082ccc777ba4f9a937931f707645e54
I20240501 14:09:43.953927 31601 tablet_copy_source_session.cc:205] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.954170 2081 tablet_copy_client.cc:523] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.954856 2205 ts_tablet_manager.cc:902] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:43.955499 2205 tablet_copy_client.cc:308] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:43.955991 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet 5b2e30cb9de14fe19273b3cef801cb67 from peer a63e0dd1ba2e4a8db8ace9718a700c32 ({username='slave'} at 127.23.191.72:59731)
I20240501 14:09:43.956089 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet 5b2e30cb9de14fe19273b3cef801cb67 from peer a63e0dd1ba2e4a8db8ace9718a700c32 at {username='slave'} at 127.23.191.72:59731: session id = a63e0dd1ba2e4a8db8ace9718a700c32-5b2e30cb9de14fe19273b3cef801cb67
I20240501 14:09:43.956954 31209 tablet_copy_source_session.cc:205] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:43.958778 2205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2e30cb9de14fe19273b3cef801cb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.960253 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:43.961041 1901 tablet_copy_client.cc:783] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:43.961205 1901 tablet_copy_client.cc:647] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:43.963299 2143 tablet_bootstrap.cc:492] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap starting.
I20240501 14:09:43.981261 2194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c082ccc777ba4f9a937931f707645e54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.984903 1901 tablet_copy_client.cc:523] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:43.986704 1901 tablet_bootstrap.cc:492] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap starting.
I20240501 14:09:44.001021 2221 ts_tablet_manager.cc:902] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:44.001030 2240 ts_tablet_manager.cc:902] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:44.001578 2240 tablet_copy_client.cc:308] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:44.001842 2221 tablet_copy_client.cc:308] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:44.002234 31601 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet 2e68be1521964ba8b864801fdf3b0df3 from peer a63e0dd1ba2e4a8db8ace9718a700c32 ({username='slave'} at 127.23.191.72:34905)
I20240501 14:09:44.002344 31601 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet 2e68be1521964ba8b864801fdf3b0df3 from peer a63e0dd1ba2e4a8db8ace9718a700c32 at {username='slave'} at 127.23.191.72:34905: session id = a63e0dd1ba2e4a8db8ace9718a700c32-2e68be1521964ba8b864801fdf3b0df3
I20240501 14:09:44.002383 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet 5d994081c3414f779ba2cae431cb1d97 from peer b934fd09590a40ada094b9715d990a00 ({username='slave'} at 127.23.191.69:54165)
I20240501 14:09:44.002483 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet 5d994081c3414f779ba2cae431cb1d97 from peer b934fd09590a40ada094b9715d990a00 at {username='slave'} at 127.23.191.69:54165: session id = b934fd09590a40ada094b9715d990a00-5d994081c3414f779ba2cae431cb1d97
I20240501 14:09:44.003265 31601 tablet_copy_source_session.cc:205] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:44.003378 31340 tablet_copy_source_session.cc:205] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:44.004343 2221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d994081c3414f779ba2cae431cb1d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.012607 2246 ts_tablet_manager.cc:902] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:44.013285 2246 tablet_copy_client.cc:308] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:44.014026 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet c0b7ede9ffaf496e9b106b5d05ed25a5 from peer a63e0dd1ba2e4a8db8ace9718a700c32 ({username='slave'} at 127.23.191.72:59731)
I20240501 14:09:44.014139 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet c0b7ede9ffaf496e9b106b5d05ed25a5 from peer a63e0dd1ba2e4a8db8ace9718a700c32 at {username='slave'} at 127.23.191.72:59731: session id = a63e0dd1ba2e4a8db8ace9718a700c32-c0b7ede9ffaf496e9b106b5d05ed25a5
I20240501 14:09:44.015061 31209 tablet_copy_source_session.cc:205] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:44.016546 2240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e68be1521964ba8b864801fdf3b0df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.016888 2194 tablet_copy_client.cc:783] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.017088 2194 tablet_copy_client.cc:647] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.020066 2204 ts_tablet_manager.cc:902] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:44.020169 2246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b7ede9ffaf496e9b106b5d05ed25a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.020596 2204 tablet_copy_client.cc:308] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:44.021088 31601 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet 72ad9c714b2b4fff94063d2838835173 from peer b934fd09590a40ada094b9715d990a00 ({username='slave'} at 127.23.191.69:46931)
I20240501 14:09:44.021184 31601 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet 72ad9c714b2b4fff94063d2838835173 from peer b934fd09590a40ada094b9715d990a00 at {username='slave'} at 127.23.191.69:46931: session id = b934fd09590a40ada094b9715d990a00-72ad9c714b2b4fff94063d2838835173
I20240501 14:09:44.022025 31601 tablet_copy_source_session.cc:205] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:44.022508 2246 tablet_copy_client.cc:783] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.022662 2246 tablet_copy_client.cc:647] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.023456 2204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ad9c714b2b4fff94063d2838835173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.024521 2240 tablet_copy_client.cc:783] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.024693 2240 tablet_copy_client.cc:647] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.024797 2194 tablet_copy_client.cc:523] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.025936 2194 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap starting.
I20240501 14:09:44.026116 2204 tablet_copy_client.cc:783] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.026305 2204 tablet_copy_client.cc:647] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.026608 2189 tablet_copy_client.cc:783] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.026783 2189 tablet_copy_client.cc:647] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.026373 2221 tablet_copy_client.cc:783] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.027115 2221 tablet_copy_client.cc:647] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.028066 1901 tablet_bootstrap.cc:492] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.028306 1901 tablet_bootstrap.cc:492] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap complete.
I20240501 14:09:44.028465 1901 ts_tablet_manager.cc:1367] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent bootstrapping tablet: real 0.042s user 0.023s sys 0.002s
I20240501 14:09:44.028939 1901 raft_consensus.cc:348] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.034176 2256 ts_tablet_manager.cc:902] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:44.034787 2256 tablet_copy_client.cc:308] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:44.035260 31601 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet 72ad9c714b2b4fff94063d2838835173 from peer a63e0dd1ba2e4a8db8ace9718a700c32 ({username='slave'} at 127.23.191.72:34905)
I20240501 14:09:44.035359 31601 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet 72ad9c714b2b4fff94063d2838835173 from peer a63e0dd1ba2e4a8db8ace9718a700c32 at {username='slave'} at 127.23.191.72:34905: session id = a63e0dd1ba2e4a8db8ace9718a700c32-72ad9c714b2b4fff94063d2838835173
I20240501 14:09:44.036240 2189 tablet_copy_client.cc:523] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.036254 31601 tablet_copy_source_session.cc:205] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:44.038014 2189 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00: Bootstrap starting.
I20240501 14:09:44.039369 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ad9c714b2b4fff94063d2838835173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.041697 2256 tablet_copy_client.cc:783] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.041883 2256 tablet_copy_client.cc:647] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.034293 2240 tablet_copy_client.cc:523] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.043128 2240 tablet_bootstrap.cc:492] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap starting.
I20240501 14:09:44.045804 2078 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.046104 2078 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap complete.
I20240501 14:09:44.046331 2078 ts_tablet_manager.cc:1367] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent bootstrapping tablet: real 0.394s user 0.026s sys 0.000s
I20240501 14:09:44.046810 2078 raft_consensus.cc:348] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.047004 2078 raft_consensus.cc:729] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63e0dd1ba2e4a8db8ace9718a700c32, State: Initialized, Role: LEARNER
I20240501 14:09:44.047142 2078 consensus_queue.cc:260] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.029068 1901 raft_consensus.cc:729] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63e0dd1ba2e4a8db8ace9718a700c32, State: Initialized, Role: LEARNER
I20240501 14:09:44.048121 1901 consensus_queue.cc:260] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.049156 2078 ts_tablet_manager.cc:1398] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:44.049371 1901 ts_tablet_manager.cc:1398] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20240501 14:09:44.049671 31471 tablet_copy_service.cc:342] P 6a6fcc099d1340c383e5a429643d5b7c: Request end of tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-5aa7bdac7bd1408cbf41d894d40aebd4 received from {username='slave'} at 127.23.191.72:40223
I20240501 14:09:44.049753 31471 tablet_copy_service.cc:433] P 6a6fcc099d1340c383e5a429643d5b7c: ending tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-5aa7bdac7bd1408cbf41d894d40aebd4 on tablet 5aa7bdac7bd1408cbf41d894d40aebd4 with peer a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:44.050107 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-bca09920a7904362bc546916e3f6f5ac received from {username='slave'} at 127.23.191.72:59731
I20240501 14:09:44.050194 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-bca09920a7904362bc546916e3f6f5ac on tablet bca09920a7904362bc546916e3f6f5ac with peer a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:44.048290 2205 tablet_copy_client.cc:783] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.050971 2205 tablet_copy_client.cc:647] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.053773 2002 tablet_copy_client.cc:523] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.055410 2205 tablet_copy_client.cc:523] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.057008 2205 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap starting.
I20240501 14:09:44.058315 1555 raft_consensus.cc:1050] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 to VOTER
I20240501 14:09:44.058835 1555 consensus_queue.cc:237] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.059326 2002 tablet_bootstrap.cc:492] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap starting.
I20240501 14:09:44.060621 31580 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.061784 31187 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.062446 970 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.063580 606 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.064427 970 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.065721 1866 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.071755 2256 tablet_copy_client.cc:523] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.071923 1927 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.072166 1927 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00: Bootstrap complete.
I20240501 14:09:44.072371 1927 ts_tablet_manager.cc:1367] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00: Time spent bootstrapping tablet: real 0.122s user 0.023s sys 0.000s
I20240501 14:09:44.074898 2256 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap starting.
I20240501 14:09:44.083137 2204 tablet_copy_client.cc:523] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.084784 2204 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00: Bootstrap starting.
I20240501 14:09:44.088240 2221 tablet_copy_client.cc:523] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.089176 2246 tablet_copy_client.cc:523] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.091796 2246 tablet_bootstrap.cc:492] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap starting.
I20240501 14:09:44.096329 2114 ts_tablet_manager.cc:902] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:44.097776 2122 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.098315 2082 ts_tablet_manager.cc:902] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:44.098291 1866 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.099162 2221 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00: Bootstrap starting.
I20240501 14:09:44.100754 606 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.101498 1927 raft_consensus.cc:348] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.101954 31187 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.102587 1927 raft_consensus.cc:729] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b934fd09590a40ada094b9715d990a00, State: Initialized, Role: LEARNER
I20240501 14:09:44.103704 2266 ts_tablet_manager.cc:902] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:44.103794 1927 consensus_queue.cc:260] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.104395 31580 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.104665 31039 catalog_manager.cc:5484] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.107993 2081 tablet_bootstrap.cc:492] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:44.108999 2233 ts_tablet_manager.cc:902] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:44.113567 605 raft_consensus.cc:1203] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:09:44.114365 2082 tablet_copy_client.cc:308] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:44.114779 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet 5d994081c3414f779ba2cae431cb1d97 from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:59967)
I20240501 14:09:44.114872 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet 5d994081c3414f779ba2cae431cb1d97 from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:59967: session id = a0caf2974041408bb32362bfc4284eaa-5d994081c3414f779ba2cae431cb1d97
I20240501 14:09:44.115748 31340 tablet_copy_source_session.cc:205] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:44.117964 2115 ts_tablet_manager.cc:902] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:44.118486 2115 tablet_copy_client.cc:308] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:44.118913 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet c0b7ede9ffaf496e9b106b5d05ed25a5 from peer 9438789dc61f4501ad1d4ceaeccee458 ({username='slave'} at 127.23.191.70:36823)
I20240501 14:09:44.119024 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet c0b7ede9ffaf496e9b106b5d05ed25a5 from peer 9438789dc61f4501ad1d4ceaeccee458 at {username='slave'} at 127.23.191.70:36823: session id = 9438789dc61f4501ad1d4ceaeccee458-c0b7ede9ffaf496e9b106b5d05ed25a5
I20240501 14:09:44.119902 31209 tablet_copy_source_session.cc:205] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:44.120714 2115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b7ede9ffaf496e9b106b5d05ed25a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.123559 2082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d994081c3414f779ba2cae431cb1d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.124641 2114 tablet_copy_client.cc:308] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:44.125735 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet b1006490ec2841b2a3365295caaf19e1 from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:42437)
I20240501 14:09:44.125833 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet b1006490ec2841b2a3365295caaf19e1 from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:42437: session id = a0caf2974041408bb32362bfc4284eaa-b1006490ec2841b2a3365295caaf19e1
I20240501 14:09:44.126637 31209 tablet_copy_source_session.cc:205] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:44.131556 31580 consensus_queue.cc:237] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.134054 31187 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:44.134510 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d265eff200b645bfac8f8906654385f6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.134831 31449 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:44.135249 1214 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:44.136040 1214 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:44.138746 2115 tablet_copy_client.cc:783] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.138913 2115 tablet_copy_client.cc:647] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.140125 2122 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap complete.
I20240501 14:09:44.140312 2122 ts_tablet_manager.cc:1367] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458: Time spent bootstrapping tablet: real 0.221s user 0.027s sys 0.000s
I20240501 14:09:44.140760 2122 raft_consensus.cc:348] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.140914 2082 tablet_copy_client.cc:783] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.140925 2122 raft_consensus.cc:729] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Initialized, Role: LEARNER
I20240501 14:09:44.141047 2082 tablet_copy_client.cc:647] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.141026 2122 consensus_queue.cc:260] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.141459 2122 ts_tablet_manager.cc:1398] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.141961 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session 9438789dc61f4501ad1d4ceaeccee458-b1006490ec2841b2a3365295caaf19e1 received from {username='slave'} at 127.23.191.70:36823
I20240501 14:09:44.142047 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session 9438789dc61f4501ad1d4ceaeccee458-b1006490ec2841b2a3365295caaf19e1 on tablet b1006490ec2841b2a3365295caaf19e1 with peer 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:44.144922 2115 tablet_copy_client.cc:523] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.146143 2233 tablet_copy_client.cc:308] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:44.146363 2115 tablet_bootstrap.cc:492] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap starting.
I20240501 14:09:44.150816 1214 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.152477 2114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1006490ec2841b2a3365295caaf19e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.152685 31187 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.154572 2114 tablet_copy_client.cc:783] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.154891 2114 tablet_copy_client.cc:647] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.156991 1760 tablet_bootstrap.cc:492] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.157428 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet bca09920a7904362bc546916e3f6f5ac from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:42437)
I20240501 14:09:44.157518 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet bca09920a7904362bc546916e3f6f5ac from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:42437: session id = a0caf2974041408bb32362bfc4284eaa-bca09920a7904362bc546916e3f6f5ac
I20240501 14:09:44.158385 31209 tablet_copy_source_session.cc:205] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:44.159063 868 raft_consensus.cc:1203] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:09:44.159253 2233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca09920a7904362bc546916e3f6f5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.161089 1760 tablet_bootstrap.cc:492] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap complete.
I20240501 14:09:44.161324 1760 ts_tablet_manager.cc:1367] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458: Time spent bootstrapping tablet: real 0.414s user 0.027s sys 0.001s
I20240501 14:09:44.161573 2233 tablet_copy_client.cc:783] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.161758 2233 tablet_copy_client.cc:647] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.162299 1760 raft_consensus.cc:348] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.164463 1760 raft_consensus.cc:729] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Initialized, Role: LEARNER
I20240501 14:09:44.164587 1760 consensus_queue.cc:260] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.169670 1760 ts_tablet_manager.cc:1398] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:44.170131 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session 9438789dc61f4501ad1d4ceaeccee458-28013d7df97943e09c27c3bb1794ee80 received from {username='slave'} at 127.23.191.70:36823
I20240501 14:09:44.170212 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session 9438789dc61f4501ad1d4ceaeccee458-28013d7df97943e09c27c3bb1794ee80 on tablet 28013d7df97943e09c27c3bb1794ee80 with peer 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:44.171823 31187 consensus_queue.cc:237] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.173414 2082 tablet_copy_client.cc:523] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 14:09:44.174124 31124 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.174755 31451 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 14:09:44.175351 31123 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b2e30cb9de14fe19273b3cef801cb67. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.175465 1798 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.175526 2082 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:44.176080 31580 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.176754 1255 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.177479 1798 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.179982 868 raft_consensus.cc:1203] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:09:44.183676 31580 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.184576 31451 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.189155 31046 catalog_manager.cc:5484] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 5 to 6, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
W20240501 14:09:44.191150 31125 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.191610 31187 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.196801 31580 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:09:44.197248 31124 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.197355 1798 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.197525 606 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.198191 31319 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:09:44.198621 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.198935 2362 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.199075 1798 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.199600 1927 ts_tablet_manager.cc:1398] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00: Time spent starting tablet: real 0.127s user 0.005s sys 0.000s
W20240501 14:09:44.199651 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1006490ec2841b2a3365295caaf19e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.200345 2266 tablet_copy_client.cc:308] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:44.201009 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session b934fd09590a40ada094b9715d990a00-a7810938b8fb406e9c6ac438f54afd03 received from {username='slave'} at 127.23.191.69:37295
I20240501 14:09:44.201062 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session b934fd09590a40ada094b9715d990a00-a7810938b8fb406e9c6ac438f54afd03 on tablet a7810938b8fb406e9c6ac438f54afd03 with peer b934fd09590a40ada094b9715d990a00
I20240501 14:09:44.200924 31580 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.203593 1817 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.203837 1817 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00: Bootstrap complete.
I20240501 14:09:44.204000 1817 ts_tablet_manager.cc:1367] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00: Time spent bootstrapping tablet: real 0.670s user 0.023s sys 0.000s
I20240501 14:09:44.207693 31319 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.208166 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet 96f5e4e25a674a99a6762757f0ab1545 from peer b934fd09590a40ada094b9715d990a00 ({username='slave'} at 127.23.191.69:37295)
I20240501 14:09:44.208263 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet 96f5e4e25a674a99a6762757f0ab1545 from peer b934fd09590a40ada094b9715d990a00 at {username='slave'} at 127.23.191.69:37295: session id = b934fd09590a40ada094b9715d990a00-96f5e4e25a674a99a6762757f0ab1545
I20240501 14:09:44.208333 2134 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.208580 2134 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00: Bootstrap complete.
I20240501 14:09:44.208732 2134 ts_tablet_manager.cc:1367] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00: Time spent bootstrapping tablet: real 0.397s user 0.025s sys 0.000s
I20240501 14:09:44.209156 2134 raft_consensus.cc:348] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.209281 2134 raft_consensus.cc:729] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b934fd09590a40ada094b9715d990a00, State: Initialized, Role: LEARNER
I20240501 14:09:44.209286 31209 tablet_copy_source_session.cc:205] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:44.209372 2134 consensus_queue.cc:260] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.199433 2362 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.210669 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f5e4e25a674a99a6762757f0ab1545. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.212703 2266 tablet_copy_client.cc:783] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.213073 2114 tablet_copy_client.cc:523] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.214512 2114 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:44.216360 1589 raft_consensus.cc:1050] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 to VOTER
I20240501 14:09:44.212821 2266 tablet_copy_client.cc:647] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.216817 1589 consensus_queue.cc:237] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.217247 1817 raft_consensus.cc:348] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.217427 1817 raft_consensus.cc:729] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b934fd09590a40ada094b9715d990a00, State: Initialized, Role: LEARNER
I20240501 14:09:44.217568 1817 consensus_queue.cc:260] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.217962 1817 ts_tablet_manager.cc:1398] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240501 14:09:44.218464 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session b934fd09590a40ada094b9715d990a00-f26d16499ab54a3e9059f6e4ff041177 received from {username='slave'} at 127.23.191.69:54165
I20240501 14:09:44.218541 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session b934fd09590a40ada094b9715d990a00-f26d16499ab54a3e9059f6e4ff041177 on tablet f26d16499ab54a3e9059f6e4ff041177 with peer b934fd09590a40ada094b9715d990a00
I20240501 14:09:44.218837 605 raft_consensus.cc:1261] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.214212 2134 ts_tablet_manager.cc:1398] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:44.220278 31471 tablet_copy_service.cc:342] P 6a6fcc099d1340c383e5a429643d5b7c: Request end of tablet copy session b934fd09590a40ada094b9715d990a00-5aa7bdac7bd1408cbf41d894d40aebd4 received from {username='slave'} at 127.23.191.69:47269
I20240501 14:09:44.220356 31471 tablet_copy_service.cc:433] P 6a6fcc099d1340c383e5a429643d5b7c: ending tablet copy session b934fd09590a40ada094b9715d990a00-5aa7bdac7bd1408cbf41d894d40aebd4 on tablet 5aa7bdac7bd1408cbf41d894d40aebd4 with peer b934fd09590a40ada094b9715d990a00
I20240501 14:09:44.222266 1590 consensus_queue.cc:1035] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.223528 1590 raft_consensus.cc:1050] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER b934fd09590a40ada094b9715d990a00 to VOTER
I20240501 14:09:44.223945 1590 consensus_queue.cc:237] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.224654 31319 raft_consensus.cc:1261] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.225706 474 raft_consensus.cc:1203] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:09:44.225756 31450 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.226598 604 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.227027 474 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.227543 31319 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.227661 31451 raft_consensus.cc:1261] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.227897 1590 consensus_queue.cc:1035] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.228116 1589 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.228271 1590 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.228350 1589 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.228577 1589 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.228822 1589 consensus_queue.cc:1035] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.231083 1589 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.233100 604 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.233505 474 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.235991 31038 catalog_manager.cc:5484] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.237174 31449 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.239409 31320 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.242482 31451 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.243754 2142 tablet_bootstrap.cc:492] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.244127 2142 tablet_bootstrap.cc:492] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:44.244462 2142 ts_tablet_manager.cc:1367] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.475s user 0.030s sys 0.005s
I20240501 14:09:44.246984 2142 raft_consensus.cc:348] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.247220 2142 raft_consensus.cc:729] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:44.247532 2142 consensus_queue.cc:260] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.255065 1589 raft_consensus.cc:1050] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 to VOTER
I20240501 14:09:44.255491 1589 consensus_queue.cc:237] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.256338 1798 raft_consensus.cc:1050] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 to VOTER
I20240501 14:09:44.256944 606 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.256886 1798 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.257019 474 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.257288 1590 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.257411 31450 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.257438 1214 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.257771 1589 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.259456 31320 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.259774 603 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.260039 605 raft_consensus.cc:1203] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:44.260138 926 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.260591 31580 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.261049 605 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
W20240501 14:09:44.261317 31124 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.261411 926 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.261948 31320 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.263226 1214 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:09:44.263306 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1006490ec2841b2a3365295caaf19e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.263398 1255 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:09:44.264153 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.265635 926 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.267104 31318 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.267859 31581 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.269891 1589 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.270390 1214 raft_consensus.cc:1050] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER b934fd09590a40ada094b9715d990a00 to VOTER
I20240501 14:09:44.270794 1214 consensus_queue.cc:237] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:44.272245 31450 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.272320 606 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.273144 2143 tablet_bootstrap.cc:492] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.273437 2143 tablet_bootstrap.cc:492] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap complete.
I20240501 14:09:44.273687 2143 ts_tablet_manager.cc:1367] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458: Time spent bootstrapping tablet: real 0.310s user 0.021s sys 0.007s
I20240501 14:09:44.274399 31318 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.274770 1590 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.274818 475 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.275386 1590 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.277273 474 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.277431 31320 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.278232 31450 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.278578 606 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.278746 1590 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.280002 2233 tablet_copy_client.cc:523] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.281412 2233 tablet_bootstrap.cc:492] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:44.287639 606 raft_consensus.cc:1203] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:09:44.289006 2266 tablet_copy_client.cc:523] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.290323 2266 tablet_bootstrap.cc:492] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00: Bootstrap starting.
I20240501 14:09:44.299553 2204 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.299852 2204 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00: Bootstrap complete.
I20240501 14:09:44.300017 2204 ts_tablet_manager.cc:1367] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00: Time spent bootstrapping tablet: real 0.215s user 0.023s sys 0.004s
I20240501 14:09:44.301348 1214 raft_consensus.cc:2932] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.300438 2204 raft_consensus.cc:348] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.301743 2204 raft_consensus.cc:729] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b934fd09590a40ada094b9715d990a00, State: Initialized, Role: LEARNER
I20240501 14:09:44.301857 2204 consensus_queue.cc:260] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.302184 2204 ts_tablet_manager.cc:1398] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:44.302659 31601 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session b934fd09590a40ada094b9715d990a00-72ad9c714b2b4fff94063d2838835173 received from {username='slave'} at 127.23.191.69:46931
I20240501 14:09:44.302759 31601 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session b934fd09590a40ada094b9715d990a00-72ad9c714b2b4fff94063d2838835173 on tablet 72ad9c714b2b4fff94063d2838835173 with peer b934fd09590a40ada094b9715d990a00
I20240501 14:09:44.303269 31450 raft_consensus.cc:2932] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.304387 606 raft_consensus.cc:2932] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.274178 2143 raft_consensus.cc:348] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.304961 2143 raft_consensus.cc:729] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Initialized, Role: LEARNER
I20240501 14:09:44.305132 2143 consensus_queue.cc:260] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.305588 1589 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.306006 2143 ts_tablet_manager.cc:1398] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458: Time spent starting tablet: real 0.032s user 0.001s sys 0.000s
I20240501 14:09:44.306716 31318 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.307832 475 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.309918 31450 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.313063 31318 consensus_queue.cc:237] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.314723 31187 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.315140 2276 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.316711 31047 catalog_manager.cc:5484] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.317010 31047 catalog_manager.cc:5484] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.317188 31047 catalog_manager.cc:5484] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.317597 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session 9438789dc61f4501ad1d4ceaeccee458-0399a419a25c4981a2ee90d76cbf59b1 received from {username='slave'} at 127.23.191.70:41145
I20240501 14:09:44.317663 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session 9438789dc61f4501ad1d4ceaeccee458-0399a419a25c4981a2ee90d76cbf59b1 on tablet 0399a419a25c4981a2ee90d76cbf59b1 with peer 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:44.314924 606 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.318409 2276 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.318626 970 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.319764 606 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.320341 382 raft_consensus.cc:1050] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c: attempting to promote NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 to VOTER
I20240501 14:09:44.320330 31187 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.320807 382 consensus_queue.cc:237] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:44.322229 31318 raft_consensus.cc:2932] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.322933 31319 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.323483 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f91909d3b1b46e9b858ce5c5b2e7975 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.323746 31046 catalog_manager.cc:5484] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 reported cstate change: config changed from index 6 to 8, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.324525 868 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.324880 2175 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.325655 2175 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.326316 31187 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.329324 2175 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.330330 31047 catalog_manager.cc:5484] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.332824 31581 consensus_queue.cc:237] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.335120 475 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.335556 31450 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.336519 31187 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 14:09:44.336870 31518 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.336983 31047 catalog_manager.cc:5484] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 reported cstate change: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.337353 1214 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.338761 1214 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.340581 31580 consensus_queue.cc:237] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.341439 31582 consensus_queue.cc:237] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:44.341671 31188 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.342088 31561 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f91909d3b1b46e9b858ce5c5b2e7975 with delete_type TABLET_DATA_TOMBSTONED (TS e7eada5032fb45c3a5e3196e2b3faaab not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34312
I20240501 14:09:44.342103 1589 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.343302 474 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.343643 2378 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.343778 604 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:44.343778 31319 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:44.344146 2362 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:44.344733 2404 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:44.345026 866 raft_consensus.cc:1203] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 14:09:44.346244 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1006490ec2841b2a3365295caaf19e1. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:44.346798 31124 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:44.346884 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.347476 1589 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.348440 2405 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.349418 31188 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.350528 31451 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.351491 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1006490ec2841b2a3365295caaf19e1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.351650 31450 raft_consensus.cc:1261] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.351804 31319 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.355321 604 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.355605 31043 catalog_manager.cc:5484] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 5 to 6, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:44.357903 31046 catalog_manager.cc:5484] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 reported cstate change: config changed from index 8 to 9, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.359453 1590 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.359791 1590 consensus_queue.cc:1035] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.361774 31451 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.365330 1214 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.367211 31581 consensus_queue.cc:237] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.369625 473 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.370117 1590 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.371909 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.372435 31450 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.374019 31449 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.374611 2378 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.375260 604 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.375712 1589 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.375682 31044 catalog_manager.cc:5484] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.377610 1589 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.378825 2430 tablet_replica.cc:330] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: stopping tablet replica
I20240501 14:09:44.379010 2175 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.379127 2430 raft_consensus.cc:2229] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.379323 2430 raft_consensus.cc:2258] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.379626 473 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.380995 31189 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.381011 31318 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.381861 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d2c173b2c32494eb16e46a51ca36e31 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.381628 868 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.382288 31449 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.384757 31040 catalog_manager.cc:5484] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32 reported cstate change: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.385028 606 raft_consensus.cc:1261] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.385411 1214 consensus_queue.cc:1035] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.391417 1214 raft_consensus.cc:2932] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.393464 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6bcd6a2f3c014d5ca0e32e16cdbc4edc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.393880 31449 raft_consensus.cc:2932] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.396772 604 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.399358 606 raft_consensus.cc:2932] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.402487 2115 tablet_bootstrap.cc:492] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.402825 2115 tablet_bootstrap.cc:492] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap complete.
I20240501 14:09:44.403163 2115 ts_tablet_manager.cc:1367] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: Time spent bootstrapping tablet: real 0.257s user 0.025s sys 0.004s
I20240501 14:09:44.403656 2115 raft_consensus.cc:348] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.403853 2115 raft_consensus.cc:729] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Initialized, Role: LEARNER
I20240501 14:09:44.403990 2115 consensus_queue.cc:260] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.404450 2115 ts_tablet_manager.cc:1398] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.404523 2142 ts_tablet_manager.cc:1398] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.160s user 0.004s sys 0.000s
I20240501 14:09:44.405006 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session 9438789dc61f4501ad1d4ceaeccee458-c0b7ede9ffaf496e9b106b5d05ed25a5 received from {username='slave'} at 127.23.191.70:36823
I20240501 14:09:44.405087 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session 9438789dc61f4501ad1d4ceaeccee458-c0b7ede9ffaf496e9b106b5d05ed25a5 on tablet c0b7ede9ffaf496e9b106b5d05ed25a5 with peer 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:44.405150 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-f8cc4d3985224178aa008b4c9de5625b received from {username='slave'} at 127.23.191.71:59967
I20240501 14:09:44.405220 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-f8cc4d3985224178aa008b4c9de5625b on tablet f8cc4d3985224178aa008b4c9de5625b with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:44.406380 2189 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.408571 2189 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00: Bootstrap complete.
I20240501 14:09:44.409265 2189 ts_tablet_manager.cc:1367] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00: Time spent bootstrapping tablet: real 0.371s user 0.026s sys 0.001s
I20240501 14:09:44.409796 2189 raft_consensus.cc:348] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.410751 2189 raft_consensus.cc:729] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b934fd09590a40ada094b9715d990a00, State: Initialized, Role: LEARNER
I20240501 14:09:44.410907 2189 consensus_queue.cc:260] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.411329 2189 ts_tablet_manager.cc:1398] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:44.412227 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session b934fd09590a40ada094b9715d990a00-5b2e30cb9de14fe19273b3cef801cb67 received from {username='slave'} at 127.23.191.69:37295
I20240501 14:09:44.412322 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session b934fd09590a40ada094b9715d990a00-5b2e30cb9de14fe19273b3cef801cb67 on tablet 5b2e30cb9de14fe19273b3cef801cb67 with peer b934fd09590a40ada094b9715d990a00
I20240501 14:09:44.412531 474 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.415599 2221 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.415859 2221 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00: Bootstrap complete.
I20240501 14:09:44.416003 2221 ts_tablet_manager.cc:1367] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00: Time spent bootstrapping tablet: real 0.317s user 0.025s sys 0.001s
I20240501 14:09:44.416400 2221 raft_consensus.cc:348] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.416517 2221 raft_consensus.cc:729] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b934fd09590a40ada094b9715d990a00, State: Initialized, Role: LEARNER
I20240501 14:09:44.416605 2221 consensus_queue.cc:260] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.422242 31299 tablet_service.cc:1514] Processing DeleteTablet for tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1 with delete_type TABLET_DATA_TOMBSTONED (TS 30e0b8849e3841ca9ab0de01e719ee4a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35058
I20240501 14:09:44.422621 31561 tablet_service.cc:1514] Processing DeleteTablet for tablet b1006490ec2841b2a3365295caaf19e1 with delete_type TABLET_DATA_TOMBSTONED (TS e7eada5032fb45c3a5e3196e2b3faaab not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34312
I20240501 14:09:44.424605 31038 catalog_manager.cc:5484] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.424887 31038 catalog_manager.cc:5484] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 7 to 8, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.427917 31581 consensus_queue.cc:237] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.428301 1589 raft_consensus.cc:1050] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER b934fd09590a40ada094b9715d990a00 to VOTER
I20240501 14:09:44.429412 31318 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:09:44.429463 31517 consensus_peers.cc:485] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c082ccc777ba4f9a937931f707645e54. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.429941 31449 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.430469 1590 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.430745 1590 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:44.431958 31518 consensus_peers.cc:485] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.433494 1589 consensus_queue.cc:237] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.434274 31299 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bcd6a2f3c014d5ca0e32e16cdbc4edc with delete_type TABLET_DATA_TOMBSTONED (TS 30e0b8849e3841ca9ab0de01e719ee4a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35058
I20240501 14:09:44.434587 31298 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d2c173b2c32494eb16e46a51ca36e31 with delete_type TABLET_DATA_TOMBSTONED (TS 30e0b8849e3841ca9ab0de01e719ee4a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35058
I20240501 14:09:44.435014 31189 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.435364 475 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.435493 31581 consensus_queue.cc:237] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.435989 1589 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.436571 2378 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.436563 31450 consensus_queue.cc:237] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
W20240501 14:09:44.437422 2082 log.cc:924] Time spent T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa: Append to log took a long time: real 0.133s user 0.004s sys 0.000s
I20240501 14:09:44.437309 475 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.438680 474 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:44.439054 2378 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:09:44.439607 31518 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.439877 606 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:44.440246 1590 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:44.440678 1590 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.441381 31319 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.441880 868 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.442462 2419 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.442457 474 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.442840 2419 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.443856 2430 ts_tablet_manager.cc:1875] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.445319 2419 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.446036 2430 ts_tablet_manager.cc:1888] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:44.446115 2430 log.cc:1196] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/wals/8f91909d3b1b46e9b858ce5c5b2e7975
I20240501 14:09:44.446622 2430 tablet_replica.cc:330] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab: stopping tablet replica
I20240501 14:09:44.446611 606 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.446753 2430 raft_consensus.cc:2229] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.446933 2430 raft_consensus.cc:2258] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.447233 31032 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a542c07a3f841b0a31051a8a37083bf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.447762 868 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.451000 31040 catalog_manager.cc:5484] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32 reported cstate change: config changed from index 5 to 6, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.452406 970 raft_consensus.cc:1050] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 to VOTER
I20240501 14:09:44.452790 970 consensus_queue.cc:237] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.453088 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d2c173b2c32494eb16e46a51ca36e31 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.453289 31033 catalog_manager.cc:4830] TS e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): tablet 8f91909d3b1b46e9b858ce5c5b2e7975 (table rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]) successfully deleted
I20240501 14:09:44.453768 31450 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.455219 31189 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.455862 2221 ts_tablet_manager.cc:1398] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00: Time spent starting tablet: real 0.040s user 0.004s sys 0.000s
I20240501 14:09:44.456025 1866 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.456279 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session b934fd09590a40ada094b9715d990a00-5d994081c3414f779ba2cae431cb1d97 received from {username='slave'} at 127.23.191.69:54165
I20240501 14:09:44.456341 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session b934fd09590a40ada094b9715d990a00-5d994081c3414f779ba2cae431cb1d97 on tablet 5d994081c3414f779ba2cae431cb1d97 with peer b934fd09590a40ada094b9715d990a00
I20240501 14:09:44.457760 2266 tablet_bootstrap.cc:492] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.458119 2266 tablet_bootstrap.cc:492] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00: Bootstrap complete.
I20240501 14:09:44.458457 606 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:09:44.458663 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ad9c714b2b4fff94063d2838835173. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.458848 1866 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.458715 2266 ts_tablet_manager.cc:1367] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00: Time spent bootstrapping tablet: real 0.168s user 0.022s sys 0.004s
I20240501 14:09:44.458881 2446 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.460515 31047 catalog_manager.cc:5484] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 reported cstate change: config changed from index 8 to 9, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.460958 31581 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.461908 1866 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.462568 474 raft_consensus.cc:1203] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:44.465301 1590 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.465721 2378 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:44.467643 31188 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.469098 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a542c07a3f841b0a31051a8a37083bf with delete_type TABLET_DATA_TOMBSTONED (TS 0e4622d80cc94f12b492bdd7597a61e1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59614
I20240501 14:09:44.469288 31429 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d2c173b2c32494eb16e46a51ca36e31 with delete_type TABLET_DATA_TOMBSTONED (TS 6a6fcc099d1340c383e5a429643d5b7c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41412
I20240501 14:09:44.470924 31044 catalog_manager.cc:5484] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.471405 473 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.471808 31318 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:44.472524 1985 tablet_bootstrap.cc:492] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.472793 1985 tablet_bootstrap.cc:492] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:44.472987 1985 ts_tablet_manager.cc:1367] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.942s user 0.030s sys 0.000s
I20240501 14:09:44.473549 1985 raft_consensus.cc:348] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.474550 31047 catalog_manager.cc:5484] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:44.476016 31450 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.477638 473 raft_consensus.cc:1203] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:09:44.479882 2194 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.480190 2194 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap complete.
I20240501 14:09:44.480377 2194 ts_tablet_manager.cc:1367] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent bootstrapping tablet: real 0.454s user 0.025s sys 0.000s
I20240501 14:09:44.482035 2442 tablet_replica.cc:330] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a: stopping tablet replica
I20240501 14:09:44.482010 2194 raft_consensus.cc:348] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.482138 2194 raft_consensus.cc:729] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63e0dd1ba2e4a8db8ace9718a700c32, State: Initialized, Role: LEARNER
I20240501 14:09:44.482249 2194 consensus_queue.cc:260] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.482524 2194 ts_tablet_manager.cc:1398] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:44.482878 2442 raft_consensus.cc:2229] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.483126 2442 raft_consensus.cc:2258] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.487639 31601 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-c082ccc777ba4f9a937931f707645e54 received from {username='slave'} at 127.23.191.72:34905
I20240501 14:09:44.487725 31601 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-c082ccc777ba4f9a937931f707645e54 on tablet c082ccc777ba4f9a937931f707645e54 with peer a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:44.489430 2144 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.489801 2144 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00: Bootstrap complete.
I20240501 14:09:44.490005 2144 ts_tablet_manager.cc:1367] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00: Time spent bootstrapping tablet: real 0.616s user 0.022s sys 0.006s
I20240501 14:09:44.490443 2144 raft_consensus.cc:348] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.490638 2144 raft_consensus.cc:729] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b934fd09590a40ada094b9715d990a00, State: Initialized, Role: LEARNER
I20240501 14:09:44.490792 2144 consensus_queue.cc:260] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.491781 2144 ts_tablet_manager.cc:1398] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:44.492269 1985 raft_consensus.cc:729] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:44.492282 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session b934fd09590a40ada094b9715d990a00-b1006490ec2841b2a3365295caaf19e1 received from {username='slave'} at 127.23.191.69:37295
I20240501 14:09:44.492374 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session b934fd09590a40ada094b9715d990a00-b1006490ec2841b2a3365295caaf19e1 on tablet b1006490ec2841b2a3365295caaf19e1 with peer b934fd09590a40ada094b9715d990a00
I20240501 14:09:44.492877 2152 tablet_bootstrap.cc:492] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.493079 473 raft_consensus.cc:1203] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:09:44.493362 31319 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.494473 31449 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:44.494771 604 raft_consensus.cc:1203] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:09:44.495453 31582 consensus_queue.cc:237] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.497202 473 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.497357 31449 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.496923 2152 tablet_bootstrap.cc:492] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:44.497622 2377 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.498153 2152 ts_tablet_manager.cc:1367] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.698s user 0.026s sys 0.000s
I20240501 14:09:44.498183 2377 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.498436 31189 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.498719 2152 raft_consensus.cc:348] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.498804 2377 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.498911 2152 raft_consensus.cc:729] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:44.499049 2152 consensus_queue.cc:260] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
W20240501 14:09:44.500854 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ad9c714b2b4fff94063d2838835173. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.502110 2152 ts_tablet_manager.cc:1398] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:44.502552 1214 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.504671 31189 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.504949 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 72ad9c714b2b4fff94063d2838835173 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.507015 31449 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.507593 31582 consensus_queue.cc:237] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.508073 31038 catalog_manager.cc:5484] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 7 to 8, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.509284 473 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
W20240501 14:09:44.509474 31518 consensus_peers.cc:485] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c082ccc777ba4f9a937931f707645e54. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.509860 31319 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.510627 868 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.511034 1590 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:44.512091 31517 consensus_peers.cc:485] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c082ccc777ba4f9a937931f707645e54. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.512252 868 raft_consensus.cc:858] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }.
Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
W20240501 14:09:44.512331 868 raft_consensus.cc:1565] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }.
Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.512715 868 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:44.492547 1985 consensus_queue.cc:260] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.514506 31449 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.514945 1590 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.515455 2377 raft_consensus.cc:1013] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: attempt to promote peer a63e0dd1ba2e4a8db8ace9718a700c32: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:09:44.515628 2374 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.516055 1985 ts_tablet_manager.cc:1398] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.043s user 0.000s sys 0.001s
I20240501 14:09:44.517885 2455 tablet_replica.cc:330] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1: stopping tablet replica
I20240501 14:09:44.518225 2455 raft_consensus.cc:2229] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.518472 2455 raft_consensus.cc:2258] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.519721 1589 raft_consensus.cc:1013] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: attempt to promote peer a63e0dd1ba2e4a8db8ace9718a700c32: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:09:44.520319 31582 consensus_queue.cc:237] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:44.522931 473 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 14:09:44.523186 2002 log.cc:924] Time spent T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20240501 14:09:44.523308 2002 tablet_bootstrap.cc:492] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.523485 1589 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:44.523557 2002 tablet_bootstrap.cc:492] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap complete.
I20240501 14:09:44.523723 2002 ts_tablet_manager.cc:1367] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: Time spent bootstrapping tablet: real 0.464s user 0.027s sys 0.000s
I20240501 14:09:44.524159 2002 raft_consensus.cc:348] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.524281 2002 raft_consensus.cc:729] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Initialized, Role: LEARNER
I20240501 14:09:44.524371 2002 consensus_queue.cc:260] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.525085 2002 ts_tablet_manager.cc:1398] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.525534 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session 9438789dc61f4501ad1d4ceaeccee458-f8cc4d3985224178aa008b4c9de5625b received from {username='slave'} at 127.23.191.70:41145
I20240501 14:09:44.525615 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session 9438789dc61f4501ad1d4ceaeccee458-f8cc4d3985224178aa008b4c9de5625b on tablet f8cc4d3985224178aa008b4c9de5625b with peer 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:44.526510 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-4603d1b1b1d441b1b0be0464e2f2c9e2 received from {username='slave'} at 127.23.191.71:59967
I20240501 14:09:44.526587 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-4603d1b1b1d441b1b0be0464e2f2c9e2 on tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:44.526983 2240 tablet_bootstrap.cc:492] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.527339 2240 tablet_bootstrap.cc:492] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap complete.
I20240501 14:09:44.527549 2240 ts_tablet_manager.cc:1367] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent bootstrapping tablet: real 0.484s user 0.027s sys 0.000s
I20240501 14:09:44.527622 31451 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:44.528085 2240 raft_consensus.cc:348] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.528254 2240 raft_consensus.cc:729] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63e0dd1ba2e4a8db8ace9718a700c32, State: Initialized, Role: LEARNER
I20240501 14:09:44.528381 2240 consensus_queue.cc:260] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.528743 2240 ts_tablet_manager.cc:1398] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.529253 31601 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-2e68be1521964ba8b864801fdf3b0df3 received from {username='slave'} at 127.23.191.72:34905
I20240501 14:09:44.529350 31601 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-2e68be1521964ba8b864801fdf3b0df3 on tablet 2e68be1521964ba8b864801fdf3b0df3 with peer a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:44.531606 2246 tablet_bootstrap.cc:492] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.531947 2246 tablet_bootstrap.cc:492] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap complete.
I20240501 14:09:44.531970 2430 ts_tablet_manager.cc:1875] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.532150 2246 ts_tablet_manager.cc:1367] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent bootstrapping tablet: real 0.440s user 0.022s sys 0.005s
I20240501 14:09:44.532562 2246 raft_consensus.cc:348] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.532676 2246 raft_consensus.cc:729] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63e0dd1ba2e4a8db8ace9718a700c32, State: Initialized, Role: LEARNER
I20240501 14:09:44.533135 1589 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:44.533811 1555 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.534688 2374 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.535108 31450 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.536789 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 72ad9c714b2b4fff94063d2838835173 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.536691 473 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.537891 31581 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.538142 31038 catalog_manager.cc:5484] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:44.522330 31189 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:44.539376 1589 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:44.536324 31188 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.539757 31039 catalog_manager.cc:5484] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.542806 31189 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.543973 1214 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.544620 2266 raft_consensus.cc:348] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.544747 2266 raft_consensus.cc:729] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b934fd09590a40ada094b9715d990a00, State: Initialized, Role: LEARNER
I20240501 14:09:44.544878 2266 consensus_queue.cc:260] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.545152 31319 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.545349 2266 ts_tablet_manager.cc:1398] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00: Time spent starting tablet: real 0.086s user 0.001s sys 0.000s
I20240501 14:09:44.545987 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session b934fd09590a40ada094b9715d990a00-96f5e4e25a674a99a6762757f0ab1545 received from {username='slave'} at 127.23.191.69:37295
I20240501 14:09:44.546067 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session b934fd09590a40ada094b9715d990a00-96f5e4e25a674a99a6762757f0ab1545 on tablet 96f5e4e25a674a99a6762757f0ab1545 with peer b934fd09590a40ada094b9715d990a00
I20240501 14:09:44.546675 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:44.547516 31449 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.548511 2455 ts_tablet_manager.cc:1875] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.550791 2455 ts_tablet_manager.cc:1888] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:44.553887 31044 catalog_manager.cc:5484] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 5 to 6, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.557157 2455 log.cc:1196] T 7a542c07a3f841b0a31051a8a37083bf P 0e4622d80cc94f12b492bdd7597a61e1: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/7a542c07a3f841b0a31051a8a37083bf
I20240501 14:09:44.558306 31031 catalog_manager.cc:4830] TS 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): tablet 7a542c07a3f841b0a31051a8a37083bf (table rebalance_test_table_0 [id=fbe6d56762d84933a7b7ddaf8c314b4f]) successfully deleted
W20240501 14:09:44.552265 2256 log.cc:924] Time spent T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240501 14:09:44.547181 868 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.559343 2256 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.559484 2442 ts_tablet_manager.cc:1875] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.559711 2256 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap complete.
I20240501 14:09:44.559916 2256 ts_tablet_manager.cc:1367] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent bootstrapping tablet: real 0.485s user 0.027s sys 0.000s
I20240501 14:09:44.561807 31582 consensus_queue.cc:237] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:44.562295 2256 raft_consensus.cc:348] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.532769 2246 consensus_queue.cc:260] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.562485 2256 raft_consensus.cc:729] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63e0dd1ba2e4a8db8ace9718a700c32, State: Initialized, Role: LEARNER
I20240501 14:09:44.562742 2246 ts_tablet_manager.cc:1398] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent starting tablet: real 0.031s user 0.001s sys 0.000s
I20240501 14:09:44.562747 2256 consensus_queue.cc:260] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.562986 2378 raft_consensus.cc:1050] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 to VOTER
I20240501 14:09:44.563235 2256 ts_tablet_manager.cc:1398] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.563409 2378 consensus_queue.cc:237] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.564231 606 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.564889 31601 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-72ad9c714b2b4fff94063d2838835173 received from {username='slave'} at 127.23.191.72:34905
I20240501 14:09:44.564957 31601 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-72ad9c714b2b4fff94063d2838835173 on tablet 72ad9c714b2b4fff94063d2838835173 with peer a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:44.565063 473 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:44.565595 31319 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.565717 868 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:09:44.565737 31517 consensus_peers.cc:485] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c082ccc777ba4f9a937931f707645e54. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.566062 2377 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:44.566331 2377 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.566572 2377 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:09:44.566813 31518 consensus_peers.cc:485] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c082ccc777ba4f9a937931f707645e54. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.568728 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-c0b7ede9ffaf496e9b106b5d05ed25a5 received from {username='slave'} at 127.23.191.72:59731
I20240501 14:09:44.568817 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-c0b7ede9ffaf496e9b106b5d05ed25a5 on tablet c0b7ede9ffaf496e9b106b5d05ed25a5 with peer a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:44.569264 31318 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.569952 2114 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.570186 2114 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:44.570379 2114 ts_tablet_manager.cc:1367] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.356s user 0.022s sys 0.004s
I20240501 14:09:44.570729 2456 tablet_replica.cc:330] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c: stopping tablet replica
I20240501 14:09:44.570982 2114 raft_consensus.cc:348] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.571089 2114 raft_consensus.cc:729] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:44.571185 2114 consensus_queue.cc:260] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.571328 31319 consensus_queue.cc:237] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.571518 2456 raft_consensus.cc:2229] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.571606 2114 ts_tablet_manager.cc:1398] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:44.571758 2456 raft_consensus.cc:2258] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.572292 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-b1006490ec2841b2a3365295caaf19e1 received from {username='slave'} at 127.23.191.71:42437
I20240501 14:09:44.572372 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-b1006490ec2841b2a3365295caaf19e1 on tablet b1006490ec2841b2a3365295caaf19e1 with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:44.572896 31038 catalog_manager.cc:5484] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 6 to 7, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.573283 604 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.573647 1866 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.574354 31189 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.574478 473 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.574841 1866 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.575335 2378 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.576035 2456 ts_tablet_manager.cc:1875] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.568744 2377 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.576905 31038 catalog_manager.cc:5484] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 9 to 10, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.579483 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ad9c714b2b4fff94063d2838835173 with delete_type TABLET_DATA_TOMBSTONED (TS 0e4622d80cc94f12b492bdd7597a61e1 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59614
I20240501 14:09:44.579675 2455 tablet_replica.cc:330] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1: stopping tablet replica
I20240501 14:09:44.579806 2455 raft_consensus.cc:2229] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.579999 2455 raft_consensus.cc:2258] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.580658 868 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.581089 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 72ad9c714b2b4fff94063d2838835173 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.582176 31450 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.582554 31449 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:44.584009 868 raft_consensus.cc:1203] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:44.584012 2374 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.584285 2374 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:44.584610 31582 consensus_queue.cc:237] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.585269 2455 ts_tablet_manager.cc:1875] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.586969 31319 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 14:09:44.587106 31518 consensus_peers.cc:485] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c082ccc777ba4f9a937931f707645e54. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.587401 2455 ts_tablet_manager.cc:1888] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:09:44.587381 1590 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:09:44.587538 31517 consensus_peers.cc:485] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c082ccc777ba4f9a937931f707645e54. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.587913 868 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.587936 1866 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.588373 2455 log.cc:1196] T 72ad9c714b2b4fff94063d2838835173 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/72ad9c714b2b4fff94063d2838835173
I20240501 14:09:44.586344 31449 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.589200 2378 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.589233 31031 catalog_manager.cc:4830] TS 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): tablet 72ad9c714b2b4fff94063d2838835173 (table rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]) successfully deleted
I20240501 14:09:44.590008 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3c8200d9c20e4140906c71c746badb94 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.590333 31189 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.592172 31471 tablet_copy_service.cc:342] P 6a6fcc099d1340c383e5a429643d5b7c: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-ad0dac76fd0c4ae6b8e0549376c98e5d received from {username='slave'} at 127.23.191.71:41729
I20240501 14:09:44.592252 31471 tablet_copy_service.cc:433] P 6a6fcc099d1340c383e5a429643d5b7c: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-ad0dac76fd0c4ae6b8e0549376c98e5d on tablet ad0dac76fd0c4ae6b8e0549376c98e5d with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:44.593163 2374 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.593631 868 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.594226 31319 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.596378 2195 tablet_bootstrap.cc:492] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.596680 2195 tablet_bootstrap.cc:492] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:44.597064 31038 catalog_manager.cc:5484] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:44.600880 31429 tablet_service.cc:1514] Processing DeleteTablet for tablet c082ccc777ba4f9a937931f707645e54 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 7 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:41412
I20240501 14:09:44.604454 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:44.605744 31298 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:44.606488 2456 ts_tablet_manager.cc:1888] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:09:44.607129 2456 log.cc:1196] T 9d2c173b2c32494eb16e46a51ca36e31 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/9d2c173b2c32494eb16e46a51ca36e31
I20240501 14:09:44.607796 31032 catalog_manager.cc:4830] TS 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): tablet 9d2c173b2c32494eb16e46a51ca36e31 (table rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]) successfully deleted
I20240501 14:09:44.608136 2456 tablet_replica.cc:330] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c: stopping tablet replica
I20240501 14:09:44.608368 2456 raft_consensus.cc:2229] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.608628 2456 raft_consensus.cc:2258] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.609283 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:44.611231 2430 ts_tablet_manager.cc:1888] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:09:44.611369 2430 log.cc:1196] T b1006490ec2841b2a3365295caaf19e1 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/wals/b1006490ec2841b2a3365295caaf19e1
I20240501 14:09:44.617228 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:44.620308 31298 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:44.627343 2195 ts_tablet_manager.cc:1367] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.772s user 0.027s sys 0.001s
I20240501 14:09:44.628252 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:44.628412 2195 raft_consensus.cc:348] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.628619 2195 raft_consensus.cc:729] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:44.628779 2195 consensus_queue.cc:260] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.629933 2195 ts_tablet_manager.cc:1398] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:44.630396 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-71241540f585474a8f62da16ed2d9874 received from {username='slave'} at 127.23.191.71:42437
I20240501 14:09:44.630475 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-71241540f585474a8f62da16ed2d9874 on tablet 71241540f585474a8f62da16ed2d9874 with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:44.632309 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c082ccc777ba4f9a937931f707645e54 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.632573 31033 catalog_manager.cc:4830] TS e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): tablet b1006490ec2841b2a3365295caaf19e1 (table rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]) successfully deleted
I20240501 14:09:44.632968 2436 raft_consensus.cc:1050] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c: attempting to promote NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 to VOTER
I20240501 14:09:44.633432 2436 consensus_queue.cc:237] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:44.633580 31449 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:44.634393 1555 raft_consensus.cc:1050] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 to VOTER
I20240501 14:09:44.634840 1555 consensus_queue.cc:237] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.634963 868 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.635372 2436 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.636351 31318 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.636978 31189 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.637217 475 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.637506 2419 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.637761 2419 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.637853 382 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.637919 2456 ts_tablet_manager.cc:1875] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.638064 473 raft_consensus.cc:1203] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:44.639118 2353 raft_consensus.cc:1050] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER b934fd09590a40ada094b9715d990a00 to VOTER
I20240501 14:09:44.639351 31188 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.639410 31450 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.639541 2353 consensus_queue.cc:237] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.639706 1866 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.639997 1866 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.640080 2456 ts_tablet_manager.cc:1888] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:09:44.640146 2456 log.cc:1196] T c082ccc777ba4f9a937931f707645e54 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/c082ccc777ba4f9a937931f707645e54
I20240501 14:09:44.640616 31032 catalog_manager.cc:4830] TS 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): tablet c082ccc777ba4f9a937931f707645e54 (table rebalance_test_table_7 [id=855c959aa35441b989d386caf8f8eded]) successfully deleted
I20240501 14:09:44.641340 606 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.641664 31320 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.641835 2354 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.642204 866 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.642663 606 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.643098 2081 tablet_bootstrap.cc:492] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.643148 970 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.643271 2404 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.643464 2081 tablet_bootstrap.cc:492] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:44.643425 970 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.643625 2081 ts_tablet_manager.cc:1367] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.536s user 0.026s sys 0.000s
I20240501 14:09:44.644110 2081 raft_consensus.cc:348] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.645517 474 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.645970 31581 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.647524 2354 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.647781 2354 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.648404 31039 catalog_manager.cc:5484] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:44.651165 31561 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c8200d9c20e4140906c71c746badb94 with delete_type TABLET_DATA_TOMBSTONED (TS e7eada5032fb45c3a5e3196e2b3faaab not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34312
I20240501 14:09:44.667531 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:44.669133 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:44.669374 2430 tablet_replica.cc:330] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab: stopping tablet replica
I20240501 14:09:44.669531 2430 raft_consensus.cc:2229] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.669720 2430 raft_consensus.cc:2258] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.675940 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:44.677851 2081 raft_consensus.cc:729] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:44.678078 2081 consensus_queue.cc:260] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.679093 2081 ts_tablet_manager.cc:1398] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.035s user 0.001s sys 0.000s
I20240501 14:09:44.680640 2205 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.681284 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-b67bd0eb45a3486492b75ca3ad3be524 received from {username='slave'} at 127.23.191.71:59967
I20240501 14:09:44.681375 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-b67bd0eb45a3486492b75ca3ad3be524 on tablet b67bd0eb45a3486492b75ca3ad3be524 with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:44.681555 2430 ts_tablet_manager.cc:1875] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.684152 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:44.684705 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:44.685292 2451 log.cc:924] Time spent T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20240501 14:09:44.693950 736 raft_consensus.cc:1203] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:09:44.694612 701 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ad9c714b2b4fff94063d2838835173 with delete_type TABLET_DATA_TOMBSTONED (TS a0caf2974041408bb32362bfc4284eaa not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44688
I20240501 14:09:44.694962 2205 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap complete.
I20240501 14:09:44.695154 2205 ts_tablet_manager.cc:1367] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent bootstrapping tablet: real 0.638s user 0.026s sys 0.001s
I20240501 14:09:44.674592 2082 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.696267 604 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.696569 2082 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:44.696777 2082 ts_tablet_manager.cc:1367] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.521s user 0.027s sys 0.000s
I20240501 14:09:44.697556 2082 raft_consensus.cc:348] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.696360 2205 raft_consensus.cc:348] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.697803 2205 raft_consensus.cc:729] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63e0dd1ba2e4a8db8ace9718a700c32, State: Initialized, Role: LEARNER
I20240501 14:09:44.697914 2205 consensus_queue.cc:260] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.698117 2430 ts_tablet_manager.cc:1888] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:44.698210 2430 log.cc:1196] T 3c8200d9c20e4140906c71c746badb94 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/wals/3c8200d9c20e4140906c71c746badb94
I20240501 14:09:44.698712 31033 catalog_manager.cc:4830] TS e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): tablet 3c8200d9c20e4140906c71c746badb94 (table rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]) successfully deleted
I20240501 14:09:44.698781 2205 ts_tablet_manager.cc:1398] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:44.699200 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-5b2e30cb9de14fe19273b3cef801cb67 received from {username='slave'} at 127.23.191.72:59731
I20240501 14:09:44.699280 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-5b2e30cb9de14fe19273b3cef801cb67 on tablet 5b2e30cb9de14fe19273b3cef801cb67 with peer a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:44.699618 2082 raft_consensus.cc:729] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:44.699757 2082 consensus_queue.cc:260] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:44.700628 2082 ts_tablet_manager.cc:1398] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:44.700954 2233 tablet_bootstrap.cc:492] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.701084 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-5d994081c3414f779ba2cae431cb1d97 received from {username='slave'} at 127.23.191.71:59967
I20240501 14:09:44.701169 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-5d994081c3414f779ba2cae431cb1d97 on tablet 5d994081c3414f779ba2cae431cb1d97 with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:44.701587 2442 ts_tablet_manager.cc:1888] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:44.701737 2442 log.cc:1196] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/b83f4cd48b4b4251a8ac4ddebdb8e4e1
I20240501 14:09:44.703802 2233 tablet_bootstrap.cc:492] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:44.703963 2233 ts_tablet_manager.cc:1367] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.423s user 0.024s sys 0.003s
I20240501 14:09:44.704782 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:44.704816 736 raft_consensus.cc:1203] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:44.704347 2233 raft_consensus.cc:348] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.709525 2233 raft_consensus.cc:729] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:44.709846 2233 consensus_queue.cc:260] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.715489 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:44.718377 604 raft_consensus.cc:1203] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:09:44.722158 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:44.723507 2233 ts_tablet_manager.cc:1398] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
I20240501 14:09:44.723956 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-bca09920a7904362bc546916e3f6f5ac received from {username='slave'} at 127.23.191.71:42437
I20240501 14:09:44.724035 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-bca09920a7904362bc546916e3f6f5ac on tablet bca09920a7904362bc546916e3f6f5ac with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:44.724074 474 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.728500 31034 catalog_manager.cc:4830] TS 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265): tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1 (table rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]) successfully deleted
I20240501 14:09:44.728808 2442 tablet_replica.cc:330] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a: stopping tablet replica
I20240501 14:09:44.729105 2442 raft_consensus.cc:2229] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.729223 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:44.729337 2442 raft_consensus.cc:2258] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.736500 2233 ts_tablet_manager.cc:902] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:44.737005 2233 tablet_copy_client.cc:308] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:44.738888 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet 5b2e30cb9de14fe19273b3cef801cb67 from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:42437)
I20240501 14:09:44.738986 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet 5b2e30cb9de14fe19273b3cef801cb67 from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:42437: session id = a0caf2974041408bb32362bfc4284eaa-5b2e30cb9de14fe19273b3cef801cb67
I20240501 14:09:44.739868 31209 tablet_copy_source_session.cc:205] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:44.741479 31187 consensus_queue.cc:237] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.743535 2276 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
W20240501 14:09:44.744505 31125 consensus_peers.cc:485] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f8bd03c05d84fdaaf138a6d0608e30e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.744779 31450 raft_consensus.cc:1261] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:44.745214 2404 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:44.745990 31450 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.746263 2205 ts_tablet_manager.cc:902] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:44.746676 2205 tablet_copy_client.cc:308] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:44.747071 31601 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet d265eff200b645bfac8f8906654385f6 from peer a63e0dd1ba2e4a8db8ace9718a700c32 ({username='slave'} at 127.23.191.72:34905)
I20240501 14:09:44.747179 31601 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet d265eff200b645bfac8f8906654385f6 from peer a63e0dd1ba2e4a8db8ace9718a700c32 at {username='slave'} at 127.23.191.72:34905: session id = a63e0dd1ba2e4a8db8ace9718a700c32-d265eff200b645bfac8f8906654385f6
I20240501 14:09:44.748060 31601 tablet_copy_source_session.cc:205] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:44.749058 2205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d265eff200b645bfac8f8906654385f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.749490 736 raft_consensus.cc:1203] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:09:44.749516 31319 raft_consensus.cc:1261] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:44.749883 2404 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:44.751168 31188 consensus_queue.cc:237] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.753089 2205 tablet_copy_client.cc:783] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.753119 475 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.753371 2205 tablet_copy_client.cc:647] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.753293 31188 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.753950 2233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2e30cb9de14fe19273b3cef801cb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.754709 2354 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.754675 2406 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.755555 31581 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.756873 2233 tablet_copy_client.cc:783] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.756903 1255 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.756759 31581 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.757059 2233 tablet_copy_client.cc:647] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.757344 475 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
W20240501 14:09:44.757498 31125 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96f5e4e25a674a99a6762757f0ab1545. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.757685 2442 ts_tablet_manager.cc:1875] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.757844 2404 raft_consensus.cc:1050] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 to VOTER
I20240501 14:09:44.758141 2404 consensus_queue.cc:237] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.759742 2442 ts_tablet_manager.cc:1888] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:44.759832 605 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.759869 2442 log.cc:1196] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/6bcd6a2f3c014d5ca0e32e16cdbc4edc
I20240501 14:09:44.759974 31044 catalog_manager.cc:5484] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.760324 2442 tablet_replica.cc:330] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a: stopping tablet replica
I20240501 14:09:44.760283 31047 catalog_manager.cc:5484] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.760443 31580 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.760669 2442 raft_consensus.cc:2229] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.760874 2442 raft_consensus.cc:2258] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.760990 2406 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.761268 2406 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.761425 604 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.762049 31450 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.762461 1255 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.762648 31034 catalog_manager.cc:4830] TS 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265): tablet 6bcd6a2f3c014d5ca0e32e16cdbc4edc (table rebalance_test_table_5 [id=dd036c60a12148d2ada73beb29aedcbf]) successfully deleted
I20240501 14:09:44.763793 475 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.764282 1255 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.767599 2406 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.768615 605 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.769923 2407 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.770515 2205 tablet_copy_client.cc:523] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.772389 475 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.772536 31449 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.772653 2205 tablet_bootstrap.cc:492] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap starting.
I20240501 14:09:44.773955 31581 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.775115 31044 catalog_manager.cc:5484] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.775190 31040 catalog_manager.cc:5484] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.777772 2233 tablet_copy_client.cc:523] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.780601 31188 consensus_queue.cc:237] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.781996 605 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.782115 2442 ts_tablet_manager.cc:1875] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.782531 2233 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:44.782595 2354 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.782843 475 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.783224 31581 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.783845 2404 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.784098 2404 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.784257 2354 raft_consensus.cc:1013] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer b934fd09590a40ada094b9715d990a00: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:09:44.786984 736 raft_consensus.cc:1203] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:44.787230 841 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ad9c714b2b4fff94063d2838835173 with delete_type TABLET_DATA_TOMBSTONED (TS a63e0dd1ba2e4a8db8ace9718a700c32 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44638
I20240501 14:09:44.788033 1255 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.789904 605 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.789973 474 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.790181 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7810938b8fb406e9c6ac438f54afd03 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.790995 31047 catalog_manager.cc:5484] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 7 to 8, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:44.794940 31298 tablet_service.cc:1514] Processing DeleteTablet for tablet a7810938b8fb406e9c6ac438f54afd03 with delete_type TABLET_DATA_TOMBSTONED (TS 30e0b8849e3841ca9ab0de01e719ee4a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35058
I20240501 14:09:44.795297 31581 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.797070 2442 ts_tablet_manager.cc:1888] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:09:44.797188 2442 log.cc:1196] T 9d2c173b2c32494eb16e46a51ca36e31 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/9d2c173b2c32494eb16e46a51ca36e31
I20240501 14:09:44.797642 2442 tablet_replica.cc:330] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a: stopping tablet replica
I20240501 14:09:44.797827 31034 catalog_manager.cc:4830] TS 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265): tablet 9d2c173b2c32494eb16e46a51ca36e31 (table rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]) successfully deleted
I20240501 14:09:44.800613 31188 consensus_queue.cc:237] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.800850 2555 tablet_replica.cc:330] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: stopping tablet replica
I20240501 14:09:44.801687 970 raft_consensus.cc:1050] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 to VOTER
I20240501 14:09:44.801723 2442 raft_consensus.cc:2229] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.801893 2442 pending_rounds.cc:62] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a: Trying to abort 1 pending ops.
I20240501 14:09:44.801764 866 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.801918 606 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:44.801957 2442 pending_rounds.cc:68] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7022889306682396672 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "a7810938b8fb406e9c6ac438f54afd03" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } } }
I20240501 14:09:44.802095 2442 raft_consensus.cc:2894] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240501 14:09:44.802052 970 consensus_queue.cc:237] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.802165 2442 raft_consensus.cc:2866] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:09:44.802209 2442 raft_consensus.cc:2258] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.802284 2406 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:44.803586 31188 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.803954 1866 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.804042 2419 raft_consensus.cc:1013] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: attempt to promote peer b934fd09590a40ada094b9715d990a00: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:09:44.804457 473 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:44.805665 31450 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.805621 606 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.806043 2276 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.806609 2404 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:44.807044 2555 raft_consensus.cc:2229] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:09:44.807288 2555 raft_consensus.cc:2258] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:09:44.808118 736 raft_consensus.cc:1203] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:09:44.809239 605 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.809116 2354 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.809576 1555 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.808548 866 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.810400 1555 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.807847 2513 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.811290 866 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.812117 2555 ts_tablet_manager.cc:1875] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.813062 605 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.813942 31188 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.813980 31450 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.816007 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7810938b8fb406e9c6ac438f54afd03 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.813161 606 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.816587 473 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.817524 31044 catalog_manager.cc:5484] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 8 to 9, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:44.819080 31047 catalog_manager.cc:5484] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 6 to 7, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
W20240501 14:09:44.819836 31034 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 0399a419a25c4981a2ee90d76cbf59b1 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 6 but the committed config has opid_index of 7
I20240501 14:09:44.821034 31318 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.821429 1255 raft_consensus.cc:1050] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 to VOTER
I20240501 14:09:44.821794 1255 consensus_queue.cc:237] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.822624 382 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.823490 604 raft_consensus.cc:1261] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.823861 2404 consensus_queue.cc:1035] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.824961 868 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.826361 31561 tablet_service.cc:1514] Processing DeleteTablet for tablet a7810938b8fb406e9c6ac438f54afd03 with delete_type TABLET_DATA_TOMBSTONED (TS e7eada5032fb45c3a5e3196e2b3faaab not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34312
I20240501 14:09:44.827311 31581 raft_consensus.cc:1261] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.829699 31318 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.830024 2555 ts_tablet_manager.cc:1888] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:44.830107 2555 log.cc:1196] T 72ad9c714b2b4fff94063d2838835173 P a63e0dd1ba2e4a8db8ace9718a700c32: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal/wals/72ad9c714b2b4fff94063d2838835173
I20240501 14:09:44.832144 473 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.833012 31319 raft_consensus.cc:1261] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.833366 31034 catalog_manager.cc:4830] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): tablet 72ad9c714b2b4fff94063d2838835173 (table rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]) successfully deleted
I20240501 14:09:44.834053 2405 consensus_queue.cc:1035] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.834223 2419 raft_consensus.cc:1050] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c: attempting to promote NON_VOTER b934fd09590a40ada094b9715d990a00 to VOTER
I20240501 14:09:44.834417 2405 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.834596 2419 consensus_queue.cc:237] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:44.836438 31319 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.836608 31044 catalog_manager.cc:5484] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.836823 2436 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.837538 31451 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.839385 868 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.839735 2436 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.841727 473 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.842083 382 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.843108 2430 tablet_replica.cc:330] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: stopping tablet replica
I20240501 14:09:44.843271 2430 raft_consensus.cc:2229] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.843453 2430 raft_consensus.cc:2258] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.844146 474 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.844467 2404 consensus_queue.cc:1035] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.846079 2404 raft_consensus.cc:1050] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER b934fd09590a40ada094b9715d990a00 to VOTER
I20240501 14:09:44.845975 31044 catalog_manager.cc:5484] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:44.846478 2404 consensus_queue.cc:237] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:44.848488 31451 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.848909 2354 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:09:44.849346 31125 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96f5e4e25a674a99a6762757f0ab1545. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.849889 474 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.850116 2362 raft_consensus.cc:1050] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 to VOTER
I20240501 14:09:44.850492 2362 consensus_queue.cc:237] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:44.850689 2354 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.852370 736 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.852566 2430 ts_tablet_manager.cc:1875] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.854041 31316 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.854521 2406 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.854780 2406 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.854840 2430 ts_tablet_manager.cc:1888] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:09:44.854897 2430 log.cc:1196] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/wals/a7810938b8fb406e9c6ac438f54afd03
I20240501 14:09:44.858129 31188 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.858513 868 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.858704 382 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.859073 1255 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:09:44.859516 2521 consensus_queue.cc:1159] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" status { last_received { term: 1 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 6 } }
I20240501 14:09:44.859481 31188 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.860231 2442 ts_tablet_manager.cc:1875] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.861228 382 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.862313 2442 ts_tablet_manager.cc:1888] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:09:44.862375 2442 log.cc:1196] T a7810938b8fb406e9c6ac438f54afd03 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/a7810938b8fb406e9c6ac438f54afd03
I20240501 14:09:44.863081 865 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.863444 473 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.865712 31188 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.865979 31449 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.866353 2205 tablet_bootstrap.cc:492] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.866869 2406 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.867190 2205 tablet_bootstrap.cc:492] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap complete.
I20240501 14:09:44.867354 2205 ts_tablet_manager.cc:1367] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent bootstrapping tablet: real 0.095s user 0.027s sys 0.000s
I20240501 14:09:44.867766 2205 raft_consensus.cc:348] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.867877 2205 raft_consensus.cc:729] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63e0dd1ba2e4a8db8ace9718a700c32, State: Initialized, Role: LEARNER
I20240501 14:09:44.867974 2205 consensus_queue.cc:260] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.867976 31318 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.869122 2205 ts_tablet_manager.cc:1398] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:44.869926 31601 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-d265eff200b645bfac8f8906654385f6 received from {username='slave'} at 127.23.191.72:34905
I20240501 14:09:44.870002 31601 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-d265eff200b645bfac8f8906654385f6 on tablet d265eff200b645bfac8f8906654385f6 with peer a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:44.871795 31039 catalog_manager.cc:5484] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32 reported cstate change: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.873483 1255 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.875363 31451 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.855419 31033 catalog_manager.cc:4830] TS e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): tablet a7810938b8fb406e9c6ac438f54afd03 (table rebalance_test_table_5 [id=dd036c60a12148d2ada73beb29aedcbf]) successfully deleted
I20240501 14:09:44.862896 31034 catalog_manager.cc:4830] TS 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265): tablet a7810938b8fb406e9c6ac438f54afd03 (table rebalance_test_table_5 [id=dd036c60a12148d2ada73beb29aedcbf]) successfully deleted
I20240501 14:09:44.878577 473 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.880465 31038 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 5aa7bdac7bd1408cbf41d894d40aebd4 with cas_config_opid_index 6: aborting the task: latest config opid_index 7; task opid_index 6
I20240501 14:09:44.882632 31040 catalog_manager.cc:5484] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.886498 31451 consensus_queue.cc:237] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:44.888777 31318 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.889209 382 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.889956 868 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.890398 382 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.892601 473 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.893081 382 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.894734 2175 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.897358 31032 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5aa7bdac7bd1408cbf41d894d40aebd4 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.897961 737 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.898499 2406 consensus_queue.cc:1037] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.898726 474 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.899744 737 raft_consensus.cc:858] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
W20240501 14:09:44.899832 737 raft_consensus.cc:1565] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Could not prepare op 'term: 1 index: 8' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.900362 737 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.901029 2205 ts_tablet_manager.cc:902] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:44.901438 2205 tablet_copy_client.cc:308] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:44.901849 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet b1006490ec2841b2a3365295caaf19e1 from peer a63e0dd1ba2e4a8db8ace9718a700c32 ({username='slave'} at 127.23.191.72:59731)
I20240501 14:09:44.901933 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet b1006490ec2841b2a3365295caaf19e1 from peer a63e0dd1ba2e4a8db8ace9718a700c32 at {username='slave'} at 127.23.191.72:59731: session id = a63e0dd1ba2e4a8db8ace9718a700c32-b1006490ec2841b2a3365295caaf19e1
I20240501 14:09:44.902302 31318 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.902814 31209 tablet_copy_source_session.cc:205] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:44.900895 868 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.905586 31040 catalog_manager.cc:5484] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 7 to 8, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.905812 2205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1006490ec2841b2a3365295caaf19e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.906615 31580 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 14:09:44.907619 31032 catalog_manager.cc:4797] TS a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): delete failed for tablet 72ad9c714b2b4fff94063d2838835173 because the tablet was not found. No further retry: Not found: Tablet not found: 72ad9c714b2b4fff94063d2838835173
I20240501 14:09:44.907793 2205 tablet_copy_client.cc:783] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:44.907932 2205 tablet_copy_client.cc:647] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:44.908634 2406 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.909741 31580 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.910255 2205 tablet_copy_client.cc:523] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:44.911116 31580 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:44.911319 2205 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap starting.
I20240501 14:09:44.912533 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aa7bdac7bd1408cbf41d894d40aebd4 with delete_type TABLET_DATA_TOMBSTONED (TS 0e4622d80cc94f12b492bdd7597a61e1 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59614
I20240501 14:09:44.912734 2455 tablet_replica.cc:330] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1: stopping tablet replica
I20240501 14:09:44.912883 2455 raft_consensus.cc:2229] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.913080 2455 raft_consensus.cc:2258] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.913651 2455 ts_tablet_manager.cc:1875] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.914674 2404 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.914817 2406 raft_consensus.cc:2932] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.915262 31318 consensus_queue.cc:237] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.915696 2455 ts_tablet_manager.cc:1888] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:09:44.915766 2455 log.cc:1196] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/5aa7bdac7bd1408cbf41d894d40aebd4
W20240501 14:09:44.917820 31254 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea66fe3d0da04d18b961927a9c988a20. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.918803 2407 raft_consensus.cc:1050] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER a0caf2974041408bb32362bfc4284eaa to VOTER
I20240501 14:09:44.918835 604 raft_consensus.cc:2932] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.919090 2407 consensus_queue.cc:237] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:44.920413 31451 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:44.920799 2513 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:44.921005 736 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.921506 31580 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:44.921957 2513 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:44.922586 735 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.923987 31316 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.924121 866 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:44.924353 868 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.925048 2405 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.926000 31040 catalog_manager.cc:5484] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.926419 2353 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.926651 2353 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.927430 31316 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.930362 2354 raft_consensus.cc:1050] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 to VOTER
I20240501 14:09:44.930764 2354 consensus_queue.cc:237] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.932636 31448 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.933044 1255 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.933669 31319 raft_consensus.cc:2932] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.933889 31316 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.935115 1255 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.936709 604 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.937131 1255 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.938100 1255 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.939247 868 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.940032 31581 raft_consensus.cc:2932] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.940210 31319 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.941323 2405 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.943215 868 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.945883 31042 catalog_manager.cc:5484] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32 reported cstate change: config changed from index 6 to 7, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.945995 31040 catalog_manager.cc:5484] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458 reported cstate change: config changed from index 4 to 5, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.938396 31031 catalog_manager.cc:4830] TS 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): tablet 5aa7bdac7bd1408cbf41d894d40aebd4 (table rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]) successfully deleted
I20240501 14:09:44.950822 735 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.952033 31447 consensus_queue.cc:237] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:44.953852 474 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:44.954123 868 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:44.954247 2419 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:44.954512 382 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:44.956111 382 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.957285 474 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.958341 865 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.960033 31040 catalog_manager.cc:5484] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 8 to 9, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:44.960892 31032 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5aa7bdac7bd1408cbf41d894d40aebd4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.962487 31298 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aa7bdac7bd1408cbf41d894d40aebd4 with delete_type TABLET_DATA_TOMBSTONED (TS 30e0b8849e3841ca9ab0de01e719ee4a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35058
I20240501 14:09:44.962388 2513 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.962675 2442 tablet_replica.cc:330] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: stopping tablet replica
I20240501 14:09:44.962824 2442 raft_consensus.cc:2229] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.962997 2442 raft_consensus.cc:2258] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.963703 2442 ts_tablet_manager.cc:1875] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.963793 1255 raft_consensus.cc:1013] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer a63e0dd1ba2e4a8db8ace9718a700c32: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:09:44.965803 2442 ts_tablet_manager.cc:1888] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:09:44.965866 2442 log.cc:1196] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/5aa7bdac7bd1408cbf41d894d40aebd4
I20240501 14:09:44.966869 31034 catalog_manager.cc:4830] TS 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265): tablet 5aa7bdac7bd1408cbf41d894d40aebd4 (table rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]) successfully deleted
I20240501 14:09:44.967525 31451 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:44.967751 31046 catalog_manager.cc:5484] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:44.971500 474 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.971570 31188 consensus_queue.cc:237] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:44.972177 2404 consensus_queue.cc:1037] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.972247 2233 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.972510 2233 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:44.972693 2233 ts_tablet_manager.cc:1367] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.190s user 0.024s sys 0.001s
I20240501 14:09:44.973191 2233 raft_consensus.cc:348] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.973320 2233 raft_consensus.cc:729] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:44.973379 735 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.973423 2233 consensus_queue.cc:260] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:44.973842 474 raft_consensus.cc:858] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
W20240501 14:09:44.973919 474 raft_consensus.cc:1565] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 8' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.972609 31189 consensus_queue.cc:237] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:44.974346 474 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:44.975028 2353 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.976464 2233 ts_tablet_manager.cc:1398] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:09:44.976613 736 raft_consensus.cc:1203] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:44.976940 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-5b2e30cb9de14fe19273b3cef801cb67 received from {username='slave'} at 127.23.191.71:42437
I20240501 14:09:44.977013 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-5b2e30cb9de14fe19273b3cef801cb67 on tablet 5b2e30cb9de14fe19273b3cef801cb67 with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:44.977284 866 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.977325 606 raft_consensus.cc:1261] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.977794 2407 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:44.978044 2407 consensus_queue.cc:1035] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.979892 31581 raft_consensus.cc:1261] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:44.980265 2354 consensus_queue.cc:1035] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:44.981501 2354 raft_consensus.cc:2932] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.982374 31581 raft_consensus.cc:2932] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.983436 606 raft_consensus.cc:2932] T 28013d7df97943e09c27c3bb1794ee80 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.984364 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28013d7df97943e09c27c3bb1794ee80 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.985257 2205 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:44.985502 2205 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap complete.
I20240501 14:09:44.985658 2205 ts_tablet_manager.cc:1367] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent bootstrapping tablet: real 0.074s user 0.027s sys 0.000s
I20240501 14:09:44.986112 2205 raft_consensus.cc:348] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.986249 2205 raft_consensus.cc:729] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63e0dd1ba2e4a8db8ace9718a700c32, State: Initialized, Role: LEARNER
I20240501 14:09:44.986348 2205 consensus_queue.cc:260] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:44.986620 2205 ts_tablet_manager.cc:1398] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:44.987210 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-b1006490ec2841b2a3365295caaf19e1 received from {username='slave'} at 127.23.191.72:59731
I20240501 14:09:44.987285 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-b1006490ec2841b2a3365295caaf19e1 on tablet b1006490ec2841b2a3365295caaf19e1 with peer a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:44.988001 31038 catalog_manager.cc:5484] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:44.990273 31298 tablet_service.cc:1514] Processing DeleteTablet for tablet 28013d7df97943e09c27c3bb1794ee80 with delete_type TABLET_DATA_TOMBSTONED (TS 30e0b8849e3841ca9ab0de01e719ee4a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35058
I20240501 14:09:44.990483 2442 tablet_replica.cc:330] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a: stopping tablet replica
I20240501 14:09:44.990597 2442 raft_consensus.cc:2229] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.990736 2442 raft_consensus.cc:2258] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.991467 2442 ts_tablet_manager.cc:1875] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.993755 2442 ts_tablet_manager.cc:1888] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:44.993638 2406 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.993820 2442 log.cc:1196] T 28013d7df97943e09c27c3bb1794ee80 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/28013d7df97943e09c27c3bb1794ee80
I20240501 14:09:44.995643 31034 catalog_manager.cc:4830] TS 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265): tablet 28013d7df97943e09c27c3bb1794ee80 (table rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]) successfully deleted
I20240501 14:09:44.995945 735 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:44.997865 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bca09920a7904362bc546916e3f6f5ac with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:44.998979 31318 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:44.999346 2354 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:44.999944 868 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.000161 31038 catalog_manager.cc:5484] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:45.003180 31318 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.014276 737 raft_consensus.cc:858] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
W20240501 14:09:45.014336 737 raft_consensus.cc:1565] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:45.014560 737 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.018849 31581 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.022430 1255 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.024020 1255 raft_consensus.cc:1050] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER a0caf2974041408bb32362bfc4284eaa to VOTER
I20240501 14:09:45.024060 737 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.024427 1255 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:45.025899 31429 tablet_service.cc:1514] Processing DeleteTablet for tablet bca09920a7904362bc546916e3f6f5ac with delete_type TABLET_DATA_TOMBSTONED (TS 6a6fcc099d1340c383e5a429643d5b7c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41412
I20240501 14:09:45.026086 2456 tablet_replica.cc:330] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c: stopping tablet replica
I20240501 14:09:45.026226 2456 raft_consensus.cc:2229] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.026364 2456 pending_rounds.cc:62] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c: Trying to abort 1 pending ops.
I20240501 14:09:45.026322 31189 consensus_queue.cc:237] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:45.026428 2456 pending_rounds.cc:68] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7022889307546595328 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "bca09920a7904362bc546916e3f6f5ac" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } } }
I20240501 14:09:45.026577 2456 raft_consensus.cc:2894] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240501 14:09:45.026641 2456 raft_consensus.cc:2866] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:09:45.026682 2456 raft_consensus.cc:2258] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.025812 31448 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.027644 866 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.028007 2362 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.029942 735 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.029911 2362 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.030777 2144 ts_tablet_manager.cc:902] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:45.031085 2362 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.031251 2144 tablet_copy_client.cc:308] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:45.031980 31601 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet c082ccc777ba4f9a937931f707645e54 from peer b934fd09590a40ada094b9715d990a00 ({username='slave'} at 127.23.191.69:46931)
I20240501 14:09:45.032089 31601 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet c082ccc777ba4f9a937931f707645e54 from peer b934fd09590a40ada094b9715d990a00 at {username='slave'} at 127.23.191.69:46931: session id = b934fd09590a40ada094b9715d990a00-c082ccc777ba4f9a937931f707645e54
I20240501 14:09:45.032173 735 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.033061 31601 tablet_copy_source_session.cc:205] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:45.033504 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bca09920a7904362bc546916e3f6f5ac with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.034734 2144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c082ccc777ba4f9a937931f707645e54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.037449 866 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.041509 31317 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.045279 31046 catalog_manager.cc:5484] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 5 to 6, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.045603 2556 raft_consensus.cc:1050] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER b934fd09590a40ada094b9715d990a00 to VOTER
I20240501 14:09:45.045984 2556 consensus_queue.cc:237] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:45.048118 31189 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.048502 1555 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.050022 31580 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.050441 2513 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.051710 474 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.052129 2513 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.055533 735 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:45.055713 606 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:45.055936 31317 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:45.055917 1798 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:45.056450 473 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:45.056349 868 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.056524 866 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:45.056797 735 raft_consensus.cc:1203] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:45.056888 1798 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:45.057109 2405 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:45.057404 1798 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:45.058182 473 raft_consensus.cc:858] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:45.058169 2406 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:09:45.058245 473 raft_consensus.cc:1565] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 9' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:45.058523 2406 raft_consensus.cc:1050] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 to VOTER
I20240501 14:09:45.058467 473 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.058895 2406 consensus_queue.cc:237] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:45.059165 31039 catalog_manager.cc:5484] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 8 to 9, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.060801 31317 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.061009 2144 tablet_copy_client.cc:783] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:45.061161 2144 tablet_copy_client.cc:647] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:45.061126 2362 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.062031 31448 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.063272 31298 tablet_service.cc:1514] Processing DeleteTablet for tablet bca09920a7904362bc546916e3f6f5ac with delete_type TABLET_DATA_TOMBSTONED (TS 30e0b8849e3841ca9ab0de01e719ee4a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35058
I20240501 14:09:45.063531 2442 tablet_replica.cc:330] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a: stopping tablet replica
I20240501 14:09:45.063655 2442 raft_consensus.cc:2229] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.063846 2442 raft_consensus.cc:2258] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.063776 865 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.064678 2442 ts_tablet_manager.cc:1875] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.065375 2404 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.065519 866 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.067474 2442 ts_tablet_manager.cc:1888] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:09:45.067561 2442 log.cc:1196] T bca09920a7904362bc546916e3f6f5ac P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/bca09920a7904362bc546916e3f6f5ac
I20240501 14:09:45.068014 31034 catalog_manager.cc:4830] TS 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265): tablet bca09920a7904362bc546916e3f6f5ac (table rebalance_test_table_6 [id=1f2efe25014e49d8b36ca467c0f40ef3]) successfully deleted
I20240501 14:09:45.068401 2404 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.068907 2404 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.071168 31448 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.071300 31043 catalog_manager.cc:5484] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 6 to 7, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:45.073051 2406 raft_consensus.cc:1013] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer b934fd09590a40ada094b9715d990a00: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:09:45.074012 473 raft_consensus.cc:858] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
W20240501 14:09:45.074069 473 raft_consensus.cc:1565] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 10' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:45.073942 31317 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.074281 473 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.077029 2405 raft_consensus.cc:1013] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer b934fd09590a40ada094b9715d990a00: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:09:45.079370 2407 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.080148 2513 raft_consensus.cc:1050] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER b934fd09590a40ada094b9715d990a00 to VOTER
I20240501 14:09:45.080530 2513 consensus_queue.cc:237] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:45.081259 2144 tablet_copy_client.cc:523] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:45.081955 865 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.082566 475 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.082607 2144 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00: Bootstrap starting.
I20240501 14:09:45.082958 2556 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.083828 737 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.084012 735 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.084379 2556 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.085042 31188 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.085569 31581 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.086504 2556 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.086513 31043 catalog_manager.cc:5484] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 9 to 10, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.088757 2556 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.090446 31317 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.095845 2276 raft_consensus.cc:1050] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER a0caf2974041408bb32362bfc4284eaa to VOTER
I20240501 14:09:45.096248 2276 consensus_queue.cc:237] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:45.096446 2406 raft_consensus.cc:1050] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER b934fd09590a40ada094b9715d990a00 to VOTER
I20240501 14:09:45.096810 2406 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:45.098053 736 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.098089 31582 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.098467 1866 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.098663 1555 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.099663 31448 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.099808 472 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:45.099835 734 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:45.100095 2556 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.100281 31317 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:45.100615 1798 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:45.100759 2353 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:45.100894 2407 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:45.102031 473 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.102566 2353 raft_consensus.cc:1050] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER b934fd09590a40ada094b9715d990a00 to VOTER
I20240501 14:09:45.102980 2353 consensus_queue.cc:237] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:45.105738 736 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.106094 31451 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.106191 31582 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.106634 2406 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.107347 2556 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.107827 31187 consensus_queue.cc:237] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:45.109189 31582 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.109436 736 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.110067 31317 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.110412 2406 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:45.110908 867 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.111312 866 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.112231 473 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.112207 31038 catalog_manager.cc:5484] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 5 to 6, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.112792 2405 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.113103 2405 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.113349 2405 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:45.113601 2405 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.113842 2405 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.114483 2405 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.115752 2405 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.116217 472 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.118611 736 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.118711 31318 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.119098 2276 raft_consensus.cc:1013] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: attempt to promote peer a0caf2974041408bb32362bfc4284eaa: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:09:45.121068 31582 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.121263 2362 raft_consensus.cc:1050] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 to VOTER
I20240501 14:09:45.121156 31040 catalog_manager.cc:5484] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 10 to 11, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.121603 2362 consensus_queue.cc:237] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:45.122540 31039 catalog_manager.cc:5484] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.115187 473 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.125231 736 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.125700 31451 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.125780 970 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.126220 31581 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.126507 475 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.127871 31582 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.128006 736 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.128172 475 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.130213 2405 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.130532 2405 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.130956 31046 catalog_manager.cc:5484] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.131346 867 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.131615 1798 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:45.131891 1798 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:45.132575 31188 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.132818 864 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.133620 970 raft_consensus.cc:1050] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER a0caf2974041408bb32362bfc4284eaa to VOTER
I20240501 14:09:45.133993 970 consensus_queue.cc:237] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:45.135383 735 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.135739 1555 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.135926 31451 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.137934 602 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:45.138692 603 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.139146 1555 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.140172 2650 ts_tablet_manager.cc:902] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
W20240501 14:09:45.140539 31034 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 5d994081c3414f779ba2cae431cb1d97 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:45.140681 2650 tablet_copy_client.cc:308] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:45.141010 603 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
W20240501 14:09:45.141253 2572 consensus_queue.cc:1159] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6a6fcc099d1340c383e5a429643d5b7c" status { last_received { term: 1 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 6 } }
I20240501 14:09:45.142050 2456 ts_tablet_manager.cc:1875] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.143673 31601 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet c082ccc777ba4f9a937931f707645e54 from peer 9438789dc61f4501ad1d4ceaeccee458 ({username='slave'} at 127.23.191.70:37219)
I20240501 14:09:45.143783 31601 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet c082ccc777ba4f9a937931f707645e54 from peer 9438789dc61f4501ad1d4ceaeccee458 at {username='slave'} at 127.23.191.70:37219: session id = 9438789dc61f4501ad1d4ceaeccee458-c082ccc777ba4f9a937931f707645e54
I20240501 14:09:45.144269 2456 ts_tablet_manager.cc:1888] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:09:45.144331 2456 log.cc:1196] T bca09920a7904362bc546916e3f6f5ac P 6a6fcc099d1340c383e5a429643d5b7c: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/bca09920a7904362bc546916e3f6f5ac
I20240501 14:09:45.144656 31601 tablet_copy_source_session.cc:205] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:45.145511 31582 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.145648 31032 catalog_manager.cc:4830] TS 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): tablet bca09920a7904362bc546916e3f6f5ac (table rebalance_test_table_6 [id=1f2efe25014e49d8b36ca467c0f40ef3]) successfully deleted
I20240501 14:09:45.145846 2650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c082ccc777ba4f9a937931f707645e54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.145933 1555 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.147481 2404 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:45.147694 2405 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:45.147981 472 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.147998 2405 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.148167 2650 tablet_copy_client.cc:783] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:45.148716 1555 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.148312 2650 tablet_copy_client.cc:647] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:45.151181 865 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:45.151149 1555 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.152707 1866 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.153420 31189 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:45.155002 474 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:45.155565 2354 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:45.155474 31448 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.155977 736 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.155992 31581 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.156569 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c0b7ede9ffaf496e9b106b5d05ed25a5 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.158167 474 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.158553 31038 catalog_manager.cc:5484] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 6 to 7, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.160574 736 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:45.161402 31189 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.161988 1798 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.164559 472 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.164774 2437 raft_consensus.cc:1050] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c: attempting to promote NON_VOTER a0caf2974041408bb32362bfc4284eaa to VOTER
I20240501 14:09:45.164752 31581 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.165410 2437 consensus_queue.cc:237] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:45.166684 735 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.167042 2436 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.167141 606 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:45.167651 602 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.167812 473 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.169100 602 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.170393 31581 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.170807 2419 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.171190 31043 catalog_manager.cc:5484] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:45.172547 31317 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.172958 868 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.174981 31429 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b7ede9ffaf496e9b106b5d05ed25a5 with delete_type TABLET_DATA_TOMBSTONED (TS 6a6fcc099d1340c383e5a429643d5b7c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41412
I20240501 14:09:45.174912 31045 catalog_manager.cc:5484] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:45.175808 31318 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:45.177357 1255 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:45.177506 2406 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:45.178517 865 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.178618 602 raft_consensus.cc:1203] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.12] Dedup: 1.11->[1.12-1.12]
I20240501 14:09:45.179733 31317 consensus_queue.cc:237] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:45.179836 2456 tablet_replica.cc:330] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c: stopping tablet replica
I20240501 14:09:45.179966 2456 raft_consensus.cc:2229] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.180157 2456 raft_consensus.cc:2258] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.180272 31318 consensus_queue.cc:237] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:45.180426 31448 consensus_queue.cc:237] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:45.181443 2456 ts_tablet_manager.cc:1875] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.181874 31187 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:45.182003 31581 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:45.182595 2174 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:45.182723 31189 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.183091 735 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.183820 2276 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:45.184095 2354 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:45.184271 31045 catalog_manager.cc:5484] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 reported cstate change: config changed from index 7 to 8, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.184525 31188 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.185245 472 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.185606 1555 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:09:45.186601 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ffdb76b2c5149a18d45bee9581e605a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:45.186600 31451 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.188163 2446 raft_consensus.cc:1050] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 to VOTER
I20240501 14:09:45.188304 2456 ts_tablet_manager.cc:1888] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:09:45.188378 2456 log.cc:1196] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/c0b7ede9ffaf496e9b106b5d05ed25a5
I20240501 14:09:45.188566 2446 consensus_queue.cc:237] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:45.189853 31318 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.189934 31448 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.190352 2453 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.190526 1589 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.191776 31578 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.191923 472 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.192488 1555 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.193836 2174 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:45.194077 31034 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 5d994081c3414f779ba2cae431cb1d97 with cas_config_opid_index 6: aborting the task: latest config opid_index 7; task opid_index 6
I20240501 14:09:45.193871 2436 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.195122 2404 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.195403 865 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.195852 2453 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.197347 1555 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.198630 31318 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.198652 602 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.200892 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1006490ec2841b2a3365295caaf19e1 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.201432 31040 catalog_manager.cc:5484] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 reported cstate change: config changed from index 11 to 12, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.201437 474 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.202096 31187 consensus_queue.cc:237] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:45.204551 736 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.206784 868 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.207448 1798 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.209033 2276 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.209581 2362 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.210588 841 tablet_service.cc:1514] Processing DeleteTablet for tablet b1006490ec2841b2a3365295caaf19e1 with delete_type TABLET_DATA_TOMBSTONED (TS a63e0dd1ba2e4a8db8ace9718a700c32 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:44638
I20240501 14:09:45.211306 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c0b7ede9ffaf496e9b106b5d05ed25a5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.211371 2555 tablet_replica.cc:330] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: stopping tablet replica
I20240501 14:09:45.211475 2555 raft_consensus.cc:2229] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:09:45.211542 2555 pending_rounds.cc:62] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: Trying to abort 1 pending ops.
I20240501 14:09:45.211594 2555 pending_rounds.cc:68] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: Aborting op as it isn't in flight: id { term: 1 index: 10 } timestamp: 7022889308258988032 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "b1006490ec2841b2a3365295caaf19e1" old_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } } new_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } } }
I20240501 14:09:45.211750 2555 raft_consensus.cc:2894] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Aborting config change with OpId 1.10: Aborted: Op aborted
I20240501 14:09:45.211800 2555 raft_consensus.cc:2866] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:09:45.211835 2555 raft_consensus.cc:2258] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:09:45.211843 868 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.212702 2555 ts_tablet_manager.cc:1875] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.213574 31187 consensus_queue.cc:237] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:45.214638 2555 ts_tablet_manager.cc:1888] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 14:09:45.214699 2555 log.cc:1196] T b1006490ec2841b2a3365295caaf19e1 P a63e0dd1ba2e4a8db8ace9718a700c32: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-7/wal/wals/b1006490ec2841b2a3365295caaf19e1
I20240501 14:09:45.215255 31034 catalog_manager.cc:4830] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): tablet b1006490ec2841b2a3365295caaf19e1 (table rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]) successfully deleted
I20240501 14:09:45.215387 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5d994081c3414f779ba2cae431cb1d97 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.215593 31448 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.215786 31577 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.215992 2405 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.216148 2362 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.216532 474 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.216774 475 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.217530 865 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.217442 735 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.218189 2404 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.220240 31318 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.220291 31042 catalog_manager.cc:5484] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 7 to 8, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.220592 2144 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:45.220815 2144 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00: Bootstrap complete.
I20240501 14:09:45.220978 2144 ts_tablet_manager.cc:1367] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00: Time spent bootstrapping tablet: real 0.138s user 0.019s sys 0.007s
I20240501 14:09:45.221361 2144 raft_consensus.cc:348] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:45.221457 2144 raft_consensus.cc:729] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b934fd09590a40ada094b9715d990a00, State: Initialized, Role: LEARNER
I20240501 14:09:45.221539 2144 consensus_queue.cc:260] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:45.221851 2144 ts_tablet_manager.cc:1398] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.222088 1798 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.222275 31601 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session b934fd09590a40ada094b9715d990a00-c082ccc777ba4f9a937931f707645e54 received from {username='slave'} at 127.23.191.69:46931
I20240501 14:09:45.222343 31601 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session b934fd09590a40ada094b9715d990a00-c082ccc777ba4f9a937931f707645e54 on tablet c082ccc777ba4f9a937931f707645e54 with peer b934fd09590a40ada094b9715d990a00
I20240501 14:09:45.223150 31043 catalog_manager.cc:5484] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32 reported cstate change: config changed from index 8 to 9, VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.224975 1798 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.225693 865 raft_consensus.cc:1203] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:45.226013 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d994081c3414f779ba2cae431cb1d97 with delete_type TABLET_DATA_TOMBSTONED (TS 0e4622d80cc94f12b492bdd7597a61e1 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59614
I20240501 14:09:45.226212 2455 tablet_replica.cc:330] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1: stopping tablet replica
I20240501 14:09:45.226334 2455 raft_consensus.cc:2229] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.226501 2455 raft_consensus.cc:2258] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.227526 31448 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.228226 31032 catalog_manager.cc:4830] TS 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): tablet c0b7ede9ffaf496e9b106b5d05ed25a5 (table rebalance_test_table_3 [id=351ff5b1095247928fbb77893616e4a9]) successfully deleted
I20240501 14:09:45.229786 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b2e30cb9de14fe19273b3cef801cb67 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.229846 31039 catalog_manager.cc:5484] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.229960 31577 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.231403 31187 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:45.232394 2362 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.233438 735 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:09:45.234194 2362 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:45.235756 606 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:09:45.235791 475 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:09:45.236443 865 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.236804 474 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.238124 2405 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:45.240005 2404 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:45.240890 31318 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.244745 1255 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.246637 735 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.247320 31317 consensus_queue.cc:237] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:45.248256 605 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.248466 601 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.249485 601 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.250511 735 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.250833 970 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.251299 31577 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.251343 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1006490ec2841b2a3365295caaf19e1 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.251791 970 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.252241 31046 catalog_manager.cc:5484] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 12 to 13, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.252926 31448 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.253249 2513 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.254843 31576 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.256562 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8cc4d3985224178aa008b4c9de5625b with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.256561 31298 tablet_service.cc:1514] Processing DeleteTablet for tablet b1006490ec2841b2a3365295caaf19e1 with delete_type TABLET_DATA_TOMBSTONED (TS 30e0b8849e3841ca9ab0de01e719ee4a not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:35058
I20240501 14:09:45.256778 2442 tablet_replica.cc:330] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a: stopping tablet replica
I20240501 14:09:45.256951 2442 raft_consensus.cc:2229] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.256811 1589 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.257030 31040 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet b1006490ec2841b2a3365295caaf19e1 with cas_config_opid_index 12: aborting the task: latest config opid_index 13; task opid_index 12
I20240501 14:09:45.257213 2442 raft_consensus.cc:2258] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.258500 2442 ts_tablet_manager.cc:1875] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.258452 865 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.258586 2556 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.258536 31318 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.259486 735 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.260452 2442 ts_tablet_manager.cc:1888] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 14:09:45.260515 2442 log.cc:1196] T b1006490ec2841b2a3365295caaf19e1 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/b1006490ec2841b2a3365295caaf19e1
I20240501 14:09:45.261159 31034 catalog_manager.cc:4830] TS 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265): tablet b1006490ec2841b2a3365295caaf19e1 (table rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]) successfully deleted
I20240501 14:09:45.261576 31187 consensus_queue.cc:237] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:45.261647 31448 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.261955 31044 catalog_manager.cc:5484] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 6 to 7, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.262058 31045 catalog_manager.cc:5484] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32 reported cstate change: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.263077 701 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b2e30cb9de14fe19273b3cef801cb67 with delete_type TABLET_DATA_TOMBSTONED (TS a0caf2974041408bb32362bfc4284eaa not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44688
I20240501 14:09:45.263530 2528 tablet_replica.cc:330] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: stopping tablet replica
I20240501 14:09:45.263854 2528 raft_consensus.cc:2229] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:09:45.264067 2528 pending_rounds.cc:62] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Trying to abort 1 pending ops.
I20240501 14:09:45.264123 2528 pending_rounds.cc:68] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7022889308580630528 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "5b2e30cb9de14fe19273b3cef801cb67" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } } }
I20240501 14:09:45.264292 2528 raft_consensus.cc:2894] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240501 14:09:45.264357 2528 raft_consensus.cc:2866] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:09:45.264400 2528 raft_consensus.cc:2258] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:09:45.265296 2455 ts_tablet_manager.cc:1875] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.266929 31576 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:45.267372 2455 ts_tablet_manager.cc:1888] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:09:45.267386 865 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:45.267444 2455 log.cc:1196] T 5d994081c3414f779ba2cae431cb1d97 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/5d994081c3414f779ba2cae431cb1d97
I20240501 14:09:45.268617 31031 catalog_manager.cc:4830] TS 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): tablet 5d994081c3414f779ba2cae431cb1d97 (table rebalance_test_table_8 [id=b01dc7a764a349c9a4a9180993e4fbc3]) successfully deleted
I20240501 14:09:45.268994 31187 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:45.269587 1255 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:45.270623 603 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:09:45.271294 473 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.271497 735 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:09:45.272464 2404 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:45.272845 2650 tablet_copy_client.cc:523] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:45.272866 2404 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:45.273152 2436 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:45.273473 2404 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:45.268123 31451 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.274210 2650 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap starting.
I20240501 14:09:45.274451 2436 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.274994 474 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:45.277664 2404 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.278676 31041 catalog_manager.cc:5484] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:45.278718 736 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.278976 31041 catalog_manager.cc:5484] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:45.280086 1255 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.282199 865 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.282954 31189 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:45.284070 735 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.284314 603 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.284348 31188 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.285315 31576 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.286052 1255 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:45.286866 31298 tablet_service.cc:1514] Processing DeleteTablet for tablet b1006490ec2841b2a3365295caaf19e1 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 12 (current committed config index is 13)) from {username='slave'} at 127.0.0.1:35058
I20240501 14:09:45.287549 31042 catalog_manager.cc:5484] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 reported cstate change: config changed from index 13 to 14, VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.287634 31317 consensus_queue.cc:237] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:45.288930 31034 catalog_manager.cc:4830] TS 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265): tablet b1006490ec2841b2a3365295caaf19e1 (table rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]) successfully deleted
I20240501 14:09:45.289402 31188 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.289686 2556 raft_consensus.cc:1050] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER a0caf2974041408bb32362bfc4284eaa to VOTER
I20240501 14:09:45.289777 2276 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:45.290069 2556 consensus_queue.cc:237] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:45.292042 31576 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.292575 970 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.292966 31448 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.293828 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b2e30cb9de14fe19273b3cef801cb67 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.293820 970 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.293967 735 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.294071 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1006490ec2841b2a3365295caaf19e1 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.294553 970 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.294597 865 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.294697 603 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.295166 970 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:45.295464 970 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:45.295374 474 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.297430 2276 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.299491 31188 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.299712 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0399a419a25c4981a2ee90d76cbf59b1 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.300499 31042 catalog_manager.cc:5484] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 9 to 10, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.300604 603 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.302960 970 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.304639 31578 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.304265 31044 catalog_manager.cc:5484] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458 reported cstate change: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.304787 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f26d16499ab54a3e9059f6e4ff041177 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.306627 31429 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b2e30cb9de14fe19273b3cef801cb67 with delete_type TABLET_DATA_TOMBSTONED (TS 6a6fcc099d1340c383e5a429643d5b7c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41412
I20240501 14:09:45.306913 31429 tablet_service.cc:1514] Processing DeleteTablet for tablet 0399a419a25c4981a2ee90d76cbf59b1 with delete_type TABLET_DATA_TOMBSTONED (TS 6a6fcc099d1340c383e5a429643d5b7c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41412
I20240501 14:09:45.307261 2456 tablet_replica.cc:330] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c: stopping tablet replica
I20240501 14:09:45.307967 2456 raft_consensus.cc:2229] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.307965 31318 consensus_queue.cc:237] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:45.308308 2456 raft_consensus.cc:2258] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.309324 865 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.309597 2456 ts_tablet_manager.cc:1875] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.311717 2456 ts_tablet_manager.cc:1888] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:09:45.311844 2456 log.cc:1196] T 5b2e30cb9de14fe19273b3cef801cb67 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/5b2e30cb9de14fe19273b3cef801cb67
I20240501 14:09:45.312361 2456 tablet_replica.cc:330] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c: stopping tablet replica
I20240501 14:09:45.312418 31032 catalog_manager.cc:4830] TS 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): tablet 5b2e30cb9de14fe19273b3cef801cb67 (table rebalance_test_table_0 [id=fbe6d56762d84933a7b7ddaf8c314b4f]) successfully deleted
I20240501 14:09:45.312613 31451 consensus_queue.cc:237] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:45.313968 736 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.313995 733 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.314359 382 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.314416 2556 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.316040 2437 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.319468 31032 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ad0dac76fd0c4ae6b8e0549376c98e5d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.319800 31188 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.319994 473 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.320170 2419 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.320809 2276 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.320878 736 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.321322 31579 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.321210 2513 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.322475 733 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.323683 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5d994081c3414f779ba2cae431cb1d97 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.323776 31188 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.324623 31042 catalog_manager.cc:5484] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.324764 31041 catalog_manager.cc:5484] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 5 to 6, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:45.328015 472 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.331350 31561 tablet_service.cc:1514] Processing DeleteTablet for tablet ad0dac76fd0c4ae6b8e0549376c98e5d with delete_type TABLET_DATA_TOMBSTONED (TS e7eada5032fb45c3a5e3196e2b3faaab not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34312
I20240501 14:09:45.331406 31318 consensus_queue.cc:237] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:45.331555 2430 tablet_replica.cc:330] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: stopping tablet replica
I20240501 14:09:45.331684 2430 raft_consensus.cc:2229] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.331727 31579 consensus_queue.cc:237] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:45.331817 2430 pending_rounds.cc:62] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Trying to abort 1 pending ops.
I20240501 14:09:45.331883 2430 pending_rounds.cc:68] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022889308836544512 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ad0dac76fd0c4ae6b8e0549376c98e5d" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } } }
I20240501 14:09:45.332017 2430 raft_consensus.cc:2894] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 14:09:45.332079 2430 raft_consensus.cc:2866] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:09:45.332123 2430 raft_consensus.cc:2258] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.332777 31046 catalog_manager.cc:5484] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 8 to 9, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.333271 603 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.333354 31451 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.333731 2453 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.333757 970 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.335539 970 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.335589 2453 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.337257 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0399a419a25c4981a2ee90d76cbf59b1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.337821 865 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.337803 603 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.338200 31450 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.338701 31041 catalog_manager.cc:5484] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:45.339109 2446 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.339623 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e68be1521964ba8b864801fdf3b0df3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.340904 868 raft_consensus.cc:1261] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.340921 31561 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d994081c3414f779ba2cae431cb1d97 with delete_type TABLET_DATA_TOMBSTONED (TS e7eada5032fb45c3a5e3196e2b3faaab not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34312
I20240501 14:09:45.340725 31047 catalog_manager.cc:5484] T 0399a419a25c4981a2ee90d76cbf59b1 P 9438789dc61f4501ad1d4ceaeccee458 reported cstate change: config changed from index 8 to 9, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.341300 1555 consensus_queue.cc:1035] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.341533 31317 consensus_queue.cc:237] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:45.342226 868 raft_consensus.cc:2932] T 0399a419a25c4981a2ee90d76cbf59b1 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.344653 31188 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.345104 2276 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.346107 865 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
W20240501 14:09:45.347810 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f91909d3b1b46e9b858ce5c5b2e7975. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:45.348985 31298 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e68be1521964ba8b864801fdf3b0df3 with delete_type TABLET_DATA_TOMBSTONED (TS 30e0b8849e3841ca9ab0de01e719ee4a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35058
I20240501 14:09:45.349225 2442 tablet_replica.cc:330] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a: stopping tablet replica
I20240501 14:09:45.349357 2442 raft_consensus.cc:2229] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.349376 603 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.349483 2442 raft_consensus.cc:2258] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.349802 1555 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.350286 2442 ts_tablet_manager.cc:1875] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.351256 2205 ts_tablet_manager.cc:902] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:45.351686 2205 tablet_copy_client.cc:308] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:45.352173 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet 6f8bd03c05d84fdaaf138a6d0608e30e from peer a63e0dd1ba2e4a8db8ace9718a700c32 ({username='slave'} at 127.23.191.72:59731)
I20240501 14:09:45.352296 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet 6f8bd03c05d84fdaaf138a6d0608e30e from peer a63e0dd1ba2e4a8db8ace9718a700c32 at {username='slave'} at 127.23.191.72:59731: session id = a63e0dd1ba2e4a8db8ace9718a700c32-6f8bd03c05d84fdaaf138a6d0608e30e
I20240501 14:09:45.353395 31209 tablet_copy_source_session.cc:205] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:45.354212 2205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8bd03c05d84fdaaf138a6d0608e30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.356072 2205 tablet_copy_client.cc:783] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:45.356191 2205 tablet_copy_client.cc:647] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:45.357339 2442 ts_tablet_manager.cc:1888] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:45.357409 2442 log.cc:1196] T 2e68be1521964ba8b864801fdf3b0df3 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/2e68be1521964ba8b864801fdf3b0df3
I20240501 14:09:45.357887 31034 catalog_manager.cc:4830] TS 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265): tablet 2e68be1521964ba8b864801fdf3b0df3 (table rebalance_test_table_3 [id=351ff5b1095247928fbb77893616e4a9]) successfully deleted
I20240501 14:09:45.361788 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet f26d16499ab54a3e9059f6e4ff041177 with delete_type TABLET_DATA_TOMBSTONED (TS 0e4622d80cc94f12b492bdd7597a61e1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59614
I20240501 14:09:45.361984 2455 tablet_replica.cc:330] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1: stopping tablet replica
I20240501 14:09:45.362119 2455 raft_consensus.cc:2229] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.362285 2455 raft_consensus.cc:2258] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.362622 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet 0399a419a25c4981a2ee90d76cbf59b1 with delete_type TABLET_DATA_TOMBSTONED (TS 0e4622d80cc94f12b492bdd7597a61e1 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59614
I20240501 14:09:45.363184 2455 ts_tablet_manager.cc:1875] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.363832 474 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.365111 2455 ts_tablet_manager.cc:1888] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:45.365175 2455 log.cc:1196] T f26d16499ab54a3e9059f6e4ff041177 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/f26d16499ab54a3e9059f6e4ff041177
I20240501 14:09:45.365545 2455 tablet_replica.cc:330] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1: stopping tablet replica
I20240501 14:09:45.365638 2455 raft_consensus.cc:2229] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.365782 2455 raft_consensus.cc:2258] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.366376 31031 catalog_manager.cc:4830] TS 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): tablet f26d16499ab54a3e9059f6e4ff041177 (table rebalance_test_table_0 [id=fbe6d56762d84933a7b7ddaf8c314b4f]) successfully deleted
I20240501 14:09:45.366779 2455 ts_tablet_manager.cc:1875] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.367249 2456 raft_consensus.cc:2229] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.367540 2456 raft_consensus.cc:2258] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.368297 2456 ts_tablet_manager.cc:1875] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.368510 2455 ts_tablet_manager.cc:1888] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:09:45.368757 2455 log.cc:1196] T 0399a419a25c4981a2ee90d76cbf59b1 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/0399a419a25c4981a2ee90d76cbf59b1
I20240501 14:09:45.369200 31031 catalog_manager.cc:4830] TS 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): tablet 0399a419a25c4981a2ee90d76cbf59b1 (table rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]) successfully deleted
I20240501 14:09:45.370554 2456 ts_tablet_manager.cc:1888] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:09:45.370635 2456 log.cc:1196] T 0399a419a25c4981a2ee90d76cbf59b1 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/0399a419a25c4981a2ee90d76cbf59b1
W20240501 14:09:45.375370 2662 consensus_queue.cc:1159] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a0caf2974041408bb32362bfc4284eaa" status { last_received { term: 1 index: 7 } last_committed_idx: 6 last_received_current_leader { term: 1 index: 7 } }
I20240501 14:09:45.376855 2528 ts_tablet_manager.cc:1875] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.378124 31189 consensus_queue.cc:237] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:45.379813 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet 0399a419a25c4981a2ee90d76cbf59b1 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:59614
W20240501 14:09:45.379850 31125 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96f5e4e25a674a99a6762757f0ab1545. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:45.379065 2528 ts_tablet_manager.cc:1888] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:09:45.380153 2528 log.cc:1196] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal/wals/5b2e30cb9de14fe19273b3cef801cb67
I20240501 14:09:45.380316 31579 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.380714 31031 catalog_manager.cc:4830] TS 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): tablet 0399a419a25c4981a2ee90d76cbf59b1 (table rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]) successfully deleted
I20240501 14:09:45.381042 2362 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.382764 2205 tablet_copy_client.cc:523] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:45.384232 2205 tablet_bootstrap.cc:492] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap starting.
I20240501 14:09:45.385537 2276 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.385668 970 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.386083 735 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.388134 31579 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.388532 602 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.389005 31032 catalog_manager.cc:4830] TS a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): tablet 5b2e30cb9de14fe19273b3cef801cb67 (table rebalance_test_table_0 [id=fbe6d56762d84933a7b7ddaf8c314b4f]) successfully deleted
I20240501 14:09:45.389789 2362 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:45.390713 31032 catalog_manager.cc:4830] TS 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): tablet 0399a419a25c4981a2ee90d76cbf59b1 (table rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]) successfully deleted
I20240501 14:09:45.392372 31046 catalog_manager.cc:5484] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 reported cstate change: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.394338 31448 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.394985 31040 catalog_manager.cc:5484] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:45.397644 31188 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:45.398643 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96f5e4e25a674a99a6762757f0ab1545 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.401394 31579 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
W20240501 14:09:45.403061 2679 consensus_queue.cc:1159] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" status { last_received { term: 1 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 5 } }
I20240501 14:09:45.403657 31318 consensus_queue.cc:237] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:45.403859 2430 ts_tablet_manager.cc:1875] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.404616 31040 catalog_manager.cc:5484] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 6 to 7, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:45.405072 31448 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.405133 735 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.405650 970 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.405943 2430 ts_tablet_manager.cc:1888] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:45.406008 2430 log.cc:1196] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/wals/ad0dac76fd0c4ae6b8e0549376c98e5d
I20240501 14:09:45.406448 2430 tablet_replica.cc:330] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: stopping tablet replica
I20240501 14:09:45.406520 31033 catalog_manager.cc:4830] TS e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): tablet ad0dac76fd0c4ae6b8e0549376c98e5d (table rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]) successfully deleted
I20240501 14:09:45.406570 2430 raft_consensus.cc:2229] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.406700 2430 pending_rounds.cc:62] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Trying to abort 1 pending ops.
I20240501 14:09:45.406749 1555 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.406729 2430 pending_rounds.cc:68] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7022889308707856384 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "5d994081c3414f779ba2cae431cb1d97" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } } }
I20240501 14:09:45.406859 2430 raft_consensus.cc:2894] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240501 14:09:45.406920 2430 raft_consensus.cc:2866] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:09:45.406957 2430 raft_consensus.cc:2258] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.408110 2650 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:45.408291 1555 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.408962 2430 ts_tablet_manager.cc:1875] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.409152 2650 tablet_bootstrap.cc:492] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap complete.
I20240501 14:09:45.409373 2650 ts_tablet_manager.cc:1367] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458: Time spent bootstrapping tablet: real 0.135s user 0.022s sys 0.004s
I20240501 14:09:45.409812 2650 raft_consensus.cc:348] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:45.409942 2650 raft_consensus.cc:729] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Initialized, Role: LEARNER
I20240501 14:09:45.410050 2650 consensus_queue.cc:260] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:45.410396 2650 ts_tablet_manager.cc:1398] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.410914 31601 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session 9438789dc61f4501ad1d4ceaeccee458-c082ccc777ba4f9a937931f707645e54 received from {username='slave'} at 127.23.191.70:37219
I20240501 14:09:45.410989 31601 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session 9438789dc61f4501ad1d4ceaeccee458-c082ccc777ba4f9a937931f707645e54 on tablet c082ccc777ba4f9a937931f707645e54 with peer 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:45.411259 2430 ts_tablet_manager.cc:1888] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:09:45.411365 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.411526 31448 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.411964 31041 catalog_manager.cc:5484] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:45.412459 474 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.412405 735 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.414498 586 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b7ede9ffaf496e9b106b5d05ed25a5 with delete_type TABLET_DATA_TOMBSTONED (TS 9438789dc61f4501ad1d4ceaeccee458 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:46552
I20240501 14:09:45.414330 2430 log.cc:1196] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/wals/5d994081c3414f779ba2cae431cb1d97
I20240501 14:09:45.414498 585 tablet_service.cc:1514] Processing DeleteTablet for tablet f8cc4d3985224178aa008b4c9de5625b with delete_type TABLET_DATA_TOMBSTONED (TS 9438789dc61f4501ad1d4ceaeccee458 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46552
I20240501 14:09:45.415067 2404 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.416021 31561 tablet_service.cc:1514] Processing DeleteTablet for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 with delete_type TABLET_DATA_TOMBSTONED (TS e7eada5032fb45c3a5e3196e2b3faaab not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34312
I20240501 14:09:45.416116 31033 catalog_manager.cc:4830] TS e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): tablet 5d994081c3414f779ba2cae431cb1d97 (table rebalance_test_table_8 [id=b01dc7a764a349c9a4a9180993e4fbc3]) successfully deleted
I20240501 14:09:45.416986 2430 tablet_replica.cc:330] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab: stopping tablet replica
I20240501 14:09:45.417167 2430 raft_consensus.cc:2229] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.417372 2430 raft_consensus.cc:2258] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.418394 472 raft_consensus.cc:1203] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:09:45.419032 2430 ts_tablet_manager.cc:1875] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.420361 474 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:45.424224 453 tablet_service.cc:1514] Processing DeleteTablet for tablet b1006490ec2841b2a3365295caaf19e1 with delete_type TABLET_DATA_TOMBSTONED (TS b934fd09590a40ada094b9715d990a00 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:48254
I20240501 14:09:45.424706 2754 tablet_replica.cc:330] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00: stopping tablet replica
I20240501 14:09:45.425083 2754 raft_consensus.cc:2229] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.425436 2754 raft_consensus.cc:2258] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.425846 31429 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f5e4e25a674a99a6762757f0ab1545 with delete_type TABLET_DATA_TOMBSTONED (TS 6a6fcc099d1340c383e5a429643d5b7c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41412
I20240501 14:09:45.422909 2430 ts_tablet_manager.cc:1888] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:45.428917 2430 log.cc:1196] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/wals/4603d1b1b1d441b1b0be0464e2f2c9e2
I20240501 14:09:45.430545 31033 catalog_manager.cc:4830] TS e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 (table rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]) successfully deleted
I20240501 14:09:45.433722 2456 tablet_replica.cc:330] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c: stopping tablet replica
I20240501 14:09:45.438776 2456 raft_consensus.cc:2229] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.439085 2456 raft_consensus.cc:2258] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.439203 2753 tablet_replica.cc:330] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: stopping tablet replica
I20240501 14:09:45.439543 2753 raft_consensus.cc:2229] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:09:45.439780 2753 raft_consensus.cc:2258] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:09:45.442747 2456 ts_tablet_manager.cc:1875] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.447865 31581 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:45.448781 2754 ts_tablet_manager.cc:1875] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.455646 2456 ts_tablet_manager.cc:1888] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:09:45.456667 2456 log.cc:1196] T 96f5e4e25a674a99a6762757f0ab1545 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/96f5e4e25a674a99a6762757f0ab1545
I20240501 14:09:45.462258 31032 catalog_manager.cc:4830] TS 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): tablet 96f5e4e25a674a99a6762757f0ab1545 (table rebalance_test_table_6 [id=1f2efe25014e49d8b36ca467c0f40ef3]) successfully deleted
I20240501 14:09:45.463953 2753 ts_tablet_manager.cc:1875] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.467839 2753 ts_tablet_manager.cc:1888] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:09:45.470901 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:45.482985 31298 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:45.487032 2754 ts_tablet_manager.cc:1888] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20240501 14:09:45.487272 2754 log.cc:1196] T b1006490ec2841b2a3365295caaf19e1 P b934fd09590a40ada094b9715d990a00: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal/wals/b1006490ec2841b2a3365295caaf19e1
I20240501 14:09:45.487874 31033 catalog_manager.cc:4830] TS b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): tablet b1006490ec2841b2a3365295caaf19e1 (table rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]) successfully deleted
I20240501 14:09:45.494854 31298 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:45.495105 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:45.496587 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:45.501055 2753 log.cc:1196] T f8cc4d3985224178aa008b4c9de5625b P 9438789dc61f4501ad1d4ceaeccee458: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal/wals/f8cc4d3985224178aa008b4c9de5625b
I20240501 14:09:45.501796 31034 catalog_manager.cc:4830] TS 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): tablet f8cc4d3985224178aa008b4c9de5625b (table rebalance_test_table_5 [id=dd036c60a12148d2ada73beb29aedcbf]) successfully deleted
I20240501 14:09:45.501878 2753 tablet_replica.cc:330] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: stopping tablet replica
I20240501 14:09:45.502115 2753 raft_consensus.cc:2229] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.502285 2753 pending_rounds.cc:62] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: Trying to abort 1 pending ops.
I20240501 14:09:45.502378 2753 pending_rounds.cc:68] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7022889307875340288 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c0b7ede9ffaf496e9b106b5d05ed25a5" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } } }
I20240501 14:09:45.502624 2753 raft_consensus.cc:2894] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240501 14:09:45.502756 2753 raft_consensus.cc:2866] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:09:45.502846 2753 raft_consensus.cc:2258] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:09:45.505978 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:45.510448 602 raft_consensus.cc:1203] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:09:45.513197 2753 ts_tablet_manager.cc:1875] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.527052 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.536512 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:45.542863 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.545171 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.546636 2753 ts_tablet_manager.cc:1888] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:09:45.546797 2753 log.cc:1196] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 9438789dc61f4501ad1d4ceaeccee458: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal/wals/c0b7ede9ffaf496e9b106b5d05ed25a5
I20240501 14:09:45.547379 31034 catalog_manager.cc:4830] TS 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): tablet c0b7ede9ffaf496e9b106b5d05ed25a5 (table rebalance_test_table_3 [id=351ff5b1095247928fbb77893616e4a9]) successfully deleted
I20240501 14:09:45.550196 31298 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:45.556538 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.560519 2205 tablet_bootstrap.cc:492] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:45.560798 2205 tablet_bootstrap.cc:492] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap complete.
I20240501 14:09:45.561048 2205 ts_tablet_manager.cc:1367] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent bootstrapping tablet: real 0.177s user 0.023s sys 0.004s
I20240501 14:09:45.561571 2205 raft_consensus.cc:348] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:45.561687 2205 raft_consensus.cc:729] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63e0dd1ba2e4a8db8ace9718a700c32, State: Initialized, Role: LEARNER
I20240501 14:09:45.561781 2205 consensus_queue.cc:260] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:45.562603 2205 ts_tablet_manager.cc:1398] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:45.564379 1866 raft_consensus.cc:1050] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER a0caf2974041408bb32362bfc4284eaa to VOTER
I20240501 14:09:45.564812 1866 consensus_queue.cc:237] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:45.565451 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-6f8bd03c05d84fdaaf138a6d0608e30e received from {username='slave'} at 127.23.191.72:59731
I20240501 14:09:45.565521 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-6f8bd03c05d84fdaaf138a6d0608e30e on tablet 6f8bd03c05d84fdaaf138a6d0608e30e with peer a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:45.566722 735 raft_consensus.cc:1261] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.567278 31448 raft_consensus.cc:1261] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.567912 1555 consensus_queue.cc:1035] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.569527 1555 consensus_queue.cc:1035] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.573191 31189 raft_consensus.cc:1261] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.573640 1555 consensus_queue.cc:1035] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.575259 2822 ts_tablet_manager.cc:902] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:45.575479 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.578619 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:45.579078 2144 ts_tablet_manager.cc:902] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:45.579566 2144 tablet_copy_client.cc:308] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:39413
I20240501 14:09:45.579980 31471 tablet_copy_service.cc:140] P 6a6fcc099d1340c383e5a429643d5b7c: Received BeginTabletCopySession request for tablet 8ffdb76b2c5149a18d45bee9581e605a from peer b934fd09590a40ada094b9715d990a00 ({username='slave'} at 127.23.191.69:47269)
I20240501 14:09:45.580089 31471 tablet_copy_service.cc:161] P 6a6fcc099d1340c383e5a429643d5b7c: Beginning new tablet copy session on tablet 8ffdb76b2c5149a18d45bee9581e605a from peer b934fd09590a40ada094b9715d990a00 at {username='slave'} at 127.23.191.69:47269: session id = b934fd09590a40ada094b9715d990a00-8ffdb76b2c5149a18d45bee9581e605a
I20240501 14:09:45.581054 31471 tablet_copy_source_session.cc:205] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:45.581864 2144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffdb76b2c5149a18d45bee9581e605a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.583981 2144 tablet_copy_client.cc:783] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:45.584107 2144 tablet_copy_client.cc:647] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:45.585721 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.601169 2822 tablet_copy_client.cc:308] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:45.601807 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet ea66fe3d0da04d18b961927a9c988a20 from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:59967)
I20240501 14:09:45.601924 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet ea66fe3d0da04d18b961927a9c988a20 from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:59967: session id = a0caf2974041408bb32362bfc4284eaa-ea66fe3d0da04d18b961927a9c988a20
I20240501 14:09:45.602862 31340 tablet_copy_source_session.cc:205] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:45.603694 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:45.605306 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.621173 2822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea66fe3d0da04d18b961927a9c988a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.624331 2822 tablet_copy_client.cc:783] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:45.624575 2822 tablet_copy_client.cc:647] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:45.624902 2404 raft_consensus.cc:1050] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER a0caf2974041408bb32362bfc4284eaa to VOTER
I20240501 14:09:45.625309 2404 consensus_queue.cc:237] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:45.627295 31318 raft_consensus.cc:1261] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.626802 31581 raft_consensus.cc:1261] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.627970 2405 consensus_queue.cc:1035] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.628229 1255 consensus_queue.cc:1035] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.631067 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.631431 1555 raft_consensus.cc:2932] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.633847 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.633883 31448 raft_consensus.cc:2932] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.634498 31040 catalog_manager.cc:5484] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:45.638015 31189 raft_consensus.cc:2932] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.646384 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.647818 2144 tablet_copy_client.cc:523] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:45.649199 2144 tablet_bootstrap.cc:492] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00: Bootstrap starting.
I20240501 14:09:45.654227 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.654332 31317 consensus_queue.cc:237] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:45.655637 31448 raft_consensus.cc:1261] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.656060 970 consensus_queue.cc:1035] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.664609 970 raft_consensus.cc:2932] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.666236 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:45.666554 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b67bd0eb45a3486492b75ca3ad3be524 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.667462 31448 raft_consensus.cc:2932] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.670506 31043 catalog_manager.cc:5484] T b67bd0eb45a3486492b75ca3ad3be524 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 5 to 6, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.671648 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:45.673449 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet b67bd0eb45a3486492b75ca3ad3be524 with delete_type TABLET_DATA_TOMBSTONED (TS 0e4622d80cc94f12b492bdd7597a61e1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59614
I20240501 14:09:45.673650 2455 tablet_replica.cc:330] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1: stopping tablet replica
I20240501 14:09:45.673768 2455 raft_consensus.cc:2229] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.673933 2455 raft_consensus.cc:2258] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.626586 734 raft_consensus.cc:1261] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.675005 2455 ts_tablet_manager.cc:1875] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.633479 735 raft_consensus.cc:2932] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.675408 1255 consensus_queue.cc:1035] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.676604 737 raft_consensus.cc:1261] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.677242 1555 consensus_queue.cc:1035] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.678277 2455 ts_tablet_manager.cc:1888] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:45.678377 2455 log.cc:1196] T b67bd0eb45a3486492b75ca3ad3be524 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/b67bd0eb45a3486492b75ca3ad3be524
I20240501 14:09:45.678411 737 raft_consensus.cc:2932] T b67bd0eb45a3486492b75ca3ad3be524 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.679100 31031 catalog_manager.cc:4830] TS 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): tablet b67bd0eb45a3486492b75ca3ad3be524 (table rebalance_test_table_6 [id=1f2efe25014e49d8b36ca467c0f40ef3]) successfully deleted
I20240501 14:09:45.681406 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.685307 1255 raft_consensus.cc:2932] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.687145 31318 raft_consensus.cc:2932] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.694667 31042 catalog_manager.cc:5484] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:45.695708 31581 raft_consensus.cc:2932] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.705286 2822 tablet_copy_client.cc:523] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:45.706830 2822 tablet_bootstrap.cc:492] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:45.722090 734 raft_consensus.cc:2932] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.728010 31189 consensus_queue.cc:237] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:45.728503 2683 raft_consensus.cc:1050] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 to VOTER
I20240501 14:09:45.729007 2683 consensus_queue.cc:237] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:45.729905 31318 raft_consensus.cc:1261] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.730669 31448 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.730834 1798 consensus_queue.cc:1035] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.731118 2453 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.732261 865 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.737344 2453 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.744097 1798 raft_consensus.cc:2932] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.745847 31298 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:45.747088 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71241540f585474a8f62da16ed2d9874 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.748057 31040 catalog_manager.cc:5484] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:45.751896 31318 raft_consensus.cc:2932] T 71241540f585474a8f62da16ed2d9874 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.754184 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.754998 31189 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:45.755632 734 raft_consensus.cc:1261] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.756042 2404 consensus_queue.cc:1035] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.757961 2453 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:45.760011 734 raft_consensus.cc:2932] T 71241540f585474a8f62da16ed2d9874 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.762295 31561 tablet_service.cc:1514] Processing DeleteTablet for tablet 71241540f585474a8f62da16ed2d9874 with delete_type TABLET_DATA_TOMBSTONED (TS e7eada5032fb45c3a5e3196e2b3faaab not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34312
I20240501 14:09:45.762497 2430 tablet_replica.cc:330] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab: stopping tablet replica
I20240501 14:09:45.762655 2430 raft_consensus.cc:2229] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.762837 2430 raft_consensus.cc:2258] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.763573 2430 ts_tablet_manager.cc:1875] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.766121 2430 ts_tablet_manager.cc:1888] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:45.766194 2430 log.cc:1196] T 71241540f585474a8f62da16ed2d9874 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/wals/71241540f585474a8f62da16ed2d9874
I20240501 14:09:45.775923 31033 catalog_manager.cc:4830] TS e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): tablet 71241540f585474a8f62da16ed2d9874 (table rebalance_test_table_7 [id=855c959aa35441b989d386caf8f8eded]) successfully deleted
I20240501 14:09:45.780716 2144 tablet_bootstrap.cc:492] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:45.781198 2144 tablet_bootstrap.cc:492] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00: Bootstrap complete.
I20240501 14:09:45.781497 2144 ts_tablet_manager.cc:1367] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00: Time spent bootstrapping tablet: real 0.132s user 0.023s sys 0.002s
I20240501 14:09:45.782053 2144 raft_consensus.cc:348] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:45.782305 2144 raft_consensus.cc:729] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b934fd09590a40ada094b9715d990a00, State: Initialized, Role: LEARNER
I20240501 14:09:45.782488 2144 consensus_queue.cc:260] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:45.782977 2453 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.783622 2144 ts_tablet_manager.cc:1398] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:45.784109 31471 tablet_copy_service.cc:342] P 6a6fcc099d1340c383e5a429643d5b7c: Request end of tablet copy session b934fd09590a40ada094b9715d990a00-8ffdb76b2c5149a18d45bee9581e605a received from {username='slave'} at 127.23.191.69:47269
I20240501 14:09:45.784201 31471 tablet_copy_service.cc:433] P 6a6fcc099d1340c383e5a429643d5b7c: ending tablet copy session b934fd09590a40ada094b9715d990a00-8ffdb76b2c5149a18d45bee9581e605a on tablet 8ffdb76b2c5149a18d45bee9581e605a with peer b934fd09590a40ada094b9715d990a00
I20240501 14:09:45.785146 31189 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.786002 31448 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.786015 31043 catalog_manager.cc:5484] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:45.789471 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:45.789935 31318 consensus_queue.cc:237] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:45.791291 734 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.791908 2556 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:45.792367 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:45.796582 1555 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.797418 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.798615 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8cc4d3985224178aa008b4c9de5625b with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.799384 737 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.801944 31040 catalog_manager.cc:5484] T f8cc4d3985224178aa008b4c9de5625b P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 7 to 8, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.805016 31448 raft_consensus.cc:1261] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:45.805454 2513 consensus_queue.cc:1035] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:45.806804 31448 raft_consensus.cc:2932] T f8cc4d3985224178aa008b4c9de5625b P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:45.808393 31561 tablet_service.cc:1514] Processing DeleteTablet for tablet f8cc4d3985224178aa008b4c9de5625b with delete_type TABLET_DATA_TOMBSTONED (TS e7eada5032fb45c3a5e3196e2b3faaab not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34312
I20240501 14:09:45.808611 2430 tablet_replica.cc:330] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab: stopping tablet replica
I20240501 14:09:45.808754 2430 raft_consensus.cc:2229] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.808976 2430 raft_consensus.cc:2258] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.809815 2430 ts_tablet_manager.cc:1875] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.811785 2430 ts_tablet_manager.cc:1888] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:09:45.811846 2430 log.cc:1196] T f8cc4d3985224178aa008b4c9de5625b P e7eada5032fb45c3a5e3196e2b3faaab: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/wals/f8cc4d3985224178aa008b4c9de5625b
I20240501 14:09:45.812328 31033 catalog_manager.cc:4830] TS e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): tablet f8cc4d3985224178aa008b4c9de5625b (table rebalance_test_table_5 [id=dd036c60a12148d2ada73beb29aedcbf]) successfully deleted
I20240501 14:09:45.815157 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.827836 865 raft_consensus.cc:1203] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:45.830827 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:45.857486 2446 raft_consensus.cc:1050] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER b934fd09590a40ada094b9715d990a00 to VOTER
I20240501 14:09:45.857980 2446 consensus_queue.cc:237] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:45.859768 474 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.860235 2683 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.861254 602 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.861907 2683 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.862596 865 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.863005 1589 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.863468 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:45.863503 31318 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:45.865932 1589 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.892082 2683 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:45.893476 865 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.893918 2683 raft_consensus.cc:1050] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 to VOTER
I20240501 14:09:45.894258 2683 consensus_queue.cc:237] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:45.895596 602 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:45.896317 472 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:45.896631 474 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:45.897507 868 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:45.899605 2453 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:45.899484 31318 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:45.899976 31047 catalog_manager.cc:5484] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 8 to 9, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:45.902246 31581 consensus_queue.cc:237] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:45.903337 2822 tablet_bootstrap.cc:492] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:45.903736 2822 tablet_bootstrap.cc:492] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:45.903919 2822 ts_tablet_manager.cc:1367] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.197s user 0.027s sys 0.000s
I20240501 14:09:45.903921 31189 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.904428 2822 raft_consensus.cc:348] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:45.904541 2822 raft_consensus.cc:729] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:45.904634 2822 consensus_queue.cc:260] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:45.905014 2822 ts_tablet_manager.cc:1398] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.905293 2453 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.905886 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-ea66fe3d0da04d18b961927a9c988a20 received from {username='slave'} at 127.23.191.71:59967
I20240501 14:09:45.905966 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-ea66fe3d0da04d18b961927a9c988a20 on tablet ea66fe3d0da04d18b961927a9c988a20 with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:45.907141 2453 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.909072 31189 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.911844 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d265eff200b645bfac8f8906654385f6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.915906 31045 catalog_manager.cc:5484] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 5 to 6, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:45.918383 868 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:45.919678 865 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:45.920485 603 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:45.921249 2446 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:45.920495 2877 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:45.922487 2446 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:45.924962 31317 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:45.925781 2877 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:45.926015 2891 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.926443 865 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:45.926877 31317 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.928419 474 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.929275 31042 catalog_manager.cc:5484] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 9 to 10, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.930125 31429 tablet_service.cc:1514] Processing DeleteTablet for tablet d265eff200b645bfac8f8906654385f6 with delete_type TABLET_DATA_TOMBSTONED (TS 6a6fcc099d1340c383e5a429643d5b7c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41412
I20240501 14:09:45.930657 2456 tablet_replica.cc:330] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: stopping tablet replica
I20240501 14:09:45.930809 2456 raft_consensus.cc:2229] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.930965 2456 raft_consensus.cc:2258] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.932937 2456 ts_tablet_manager.cc:1875] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.935220 2456 ts_tablet_manager.cc:1888] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:45.935290 2456 log.cc:1196] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/d265eff200b645bfac8f8906654385f6
I20240501 14:09:45.935860 31032 catalog_manager.cc:4830] TS 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): tablet d265eff200b645bfac8f8906654385f6 (table rebalance_test_table_8 [id=b01dc7a764a349c9a4a9180993e4fbc3]) successfully deleted
I20240501 14:09:45.936581 31317 consensus_queue.cc:237] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:45.937441 603 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.937316 865 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.938040 602 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.938383 2556 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:09:45.939389 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c8200d9c20e4140906c71c746badb94. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:45.940316 31189 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:45.941041 2556 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:45.941493 1866 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:45.941574 31581 consensus_queue.cc:237] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:45.941887 603 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:45.942931 602 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:45.943349 2877 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:45.943493 474 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:45.943885 2877 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:45.946190 31045 catalog_manager.cc:5484] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 6 to 7, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:45.946939 865 raft_consensus.cc:1261] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:45.946835 2877 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.947271 2683 consensus_queue.cc:1035] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:45.947938 31189 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:45.948134 868 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.949867 31043 catalog_manager.cc:5484] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 10 to 11, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:45.950307 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c082ccc777ba4f9a937931f707645e54 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:45.950332 474 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.951792 603 raft_consensus.cc:2932] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:45.956118 31298 tablet_service.cc:1514] Processing DeleteTablet for tablet c082ccc777ba4f9a937931f707645e54 with delete_type TABLET_DATA_TOMBSTONED (TS 30e0b8849e3841ca9ab0de01e719ee4a not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:35058
I20240501 14:09:45.956482 2919 tablet_replica.cc:330] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a: stopping tablet replica
I20240501 14:09:45.956673 2919 raft_consensus.cc:2229] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.957028 2919 raft_consensus.cc:2258] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.957749 2919 ts_tablet_manager.cc:1875] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:45.959041 2205 ts_tablet_manager.cc:902] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:45.959712 2205 tablet_copy_client.cc:308] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:45.960048 2919 ts_tablet_manager.cc:1888] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 14:09:45.960194 2919 log.cc:1196] T c082ccc777ba4f9a937931f707645e54 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/c082ccc777ba4f9a937931f707645e54
I20240501 14:09:45.960474 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet 8f91909d3b1b46e9b858ce5c5b2e7975 from peer a63e0dd1ba2e4a8db8ace9718a700c32 ({username='slave'} at 127.23.191.72:44631)
I20240501 14:09:45.960568 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet 8f91909d3b1b46e9b858ce5c5b2e7975 from peer a63e0dd1ba2e4a8db8ace9718a700c32 at {username='slave'} at 127.23.191.72:44631: session id = a63e0dd1ba2e4a8db8ace9718a700c32-8f91909d3b1b46e9b858ce5c5b2e7975
I20240501 14:09:45.961467 31340 tablet_copy_source_session.cc:205] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:45.962000 31034 catalog_manager.cc:4830] TS 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265): tablet c082ccc777ba4f9a937931f707645e54 (table rebalance_test_table_7 [id=855c959aa35441b989d386caf8f8eded]) successfully deleted
I20240501 14:09:45.962344 2205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f91909d3b1b46e9b858ce5c5b2e7975. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.964416 2205 tablet_copy_client.cc:783] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:45.964546 2205 tablet_copy_client.cc:647] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:45.971242 2205 tablet_copy_client.cc:523] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:45.972594 2205 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap starting.
I20240501 14:09:45.972529 31189 consensus_queue.cc:237] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:45.972703 31581 consensus_queue.cc:237] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:45.974052 868 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:45.974709 474 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:45.974949 2404 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:45.975092 1589 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:45.976703 474 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:45.977398 31448 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:45.977911 2404 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 14:09:45.978406 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ad9c714b2b4fff94063d2838835173. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:45.978641 2683 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:45.980890 31581 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:45.981279 2354 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:45.982967 2406 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.983311 31581 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.985780 31047 catalog_manager.cc:5484] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 10 to 11. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.986619 868 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.990216 2683 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.992662 472 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:45.994727 31189 consensus_queue.cc:237] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:45.995898 31448 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:45.997431 868 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:45.997792 2354 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:45.998327 474 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:45.998694 734 raft_consensus.cc:1203] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:45.999430 2354 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.001446 472 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:46.002179 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b2e30cb9de14fe19273b3cef801cb67 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:46.002355 2404 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:46.003346 868 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.004388 2205 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:46.004338 31047 catalog_manager.cc:5484] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 10 to 11, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:46.004792 2205 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap complete.
I20240501 14:09:46.005062 2205 ts_tablet_manager.cc:1367] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.000s
I20240501 14:09:46.005313 31040 catalog_manager.cc:5484] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 11 to 12, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.005753 2205 raft_consensus.cc:348] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:46.007848 2205 raft_consensus.cc:729] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63e0dd1ba2e4a8db8ace9718a700c32, State: Initialized, Role: LEARNER
I20240501 14:09:46.008548 2205 consensus_queue.cc:260] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:46.008967 2205 ts_tablet_manager.cc:1398] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:46.008996 31561 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b2e30cb9de14fe19273b3cef801cb67 with delete_type TABLET_DATA_TOMBSTONED (TS e7eada5032fb45c3a5e3196e2b3faaab not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:34312
I20240501 14:09:46.009208 2430 tablet_replica.cc:330] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab: stopping tablet replica
I20240501 14:09:46.009339 2430 raft_consensus.cc:2229] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.009390 31189 consensus_queue.cc:237] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:46.009593 2430 raft_consensus.cc:2258] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.010380 2430 ts_tablet_manager.cc:1875] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:46.010668 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-8f91909d3b1b46e9b858ce5c5b2e7975 received from {username='slave'} at 127.23.191.72:44631
I20240501 14:09:46.010752 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-8f91909d3b1b46e9b858ce5c5b2e7975 on tablet 8f91909d3b1b46e9b858ce5c5b2e7975 with peer a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:46.012326 868 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:09:46.012567 2430 ts_tablet_manager.cc:1888] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 14:09:46.012634 2430 log.cc:1196] T 5b2e30cb9de14fe19273b3cef801cb67 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/wals/5b2e30cb9de14fe19273b3cef801cb67
I20240501 14:09:46.012660 2404 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 14:09:46.013542 31123 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.013998 474 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:09:46.014223 2404 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:46.014376 1798 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:46.016041 865 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:46.016487 31031 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5b2e30cb9de14fe19273b3cef801cb67 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:09:46.017131 31039 catalog_manager.cc:5484] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 12 to 13, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.017994 472 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.018963 2957 ts_tablet_manager.cc:902] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:46.019977 2957 tablet_copy_client.cc:308] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:46.020061 474 raft_consensus.cc:1203] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.11->[1.12-1.13] Dedup: 1.12->[1.13-1.13]
I20240501 14:09:46.020395 474 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:46.020573 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet 96f5e4e25a674a99a6762757f0ab1545 from peer 9438789dc61f4501ad1d4ceaeccee458 ({username='slave'} at 127.23.191.70:36823)
I20240501 14:09:46.020655 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet 96f5e4e25a674a99a6762757f0ab1545 from peer 9438789dc61f4501ad1d4ceaeccee458 at {username='slave'} at 127.23.191.70:36823: session id = 9438789dc61f4501ad1d4ceaeccee458-96f5e4e25a674a99a6762757f0ab1545
I20240501 14:09:46.021544 31209 tablet_copy_source_session.cc:205] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:46.024225 31033 catalog_manager.cc:4830] TS e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): tablet 5b2e30cb9de14fe19273b3cef801cb67 (table rebalance_test_table_0 [id=fbe6d56762d84933a7b7ddaf8c314b4f]) successfully deleted
I20240501 14:09:46.034576 31189 consensus_queue.cc:237] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:46.036058 602 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 14:09:46.036099 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7810938b8fb406e9c6ac438f54afd03. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.036119 474 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:46.036412 2406 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:46.036599 2354 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:46.040971 2957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f5e4e25a674a99a6762757f0ab1545. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.043503 2957 tablet_copy_client.cc:783] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:46.044736 2957 tablet_copy_client.cc:647] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:46.046066 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.048657 31298 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.055054 2957 tablet_copy_client.cc:523] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:46.055472 1798 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:46.055897 474 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:46.056602 2957 tablet_bootstrap.cc:492] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap starting.
I20240501 14:09:46.058696 31045 catalog_manager.cc:5484] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 9 to 10, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:46.063009 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.063110 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.068874 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.080873 31581 consensus_queue.cc:237] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:46.082178 868 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.082602 31448 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:09:46.082628 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e68be1521964ba8b864801fdf3b0df3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.083016 2877 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.083202 2453 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.085139 2877 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:46.085485 868 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:46.086416 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:46.087781 31450 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:46.088203 31042 catalog_manager.cc:5484] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 6 to 7, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.114697 602 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:46.120373 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.124271 474 raft_consensus.cc:1203] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:46.127363 2957 tablet_bootstrap.cc:492] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:46.127911 2957 tablet_bootstrap.cc:492] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap complete.
I20240501 14:09:46.128100 2957 ts_tablet_manager.cc:1367] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458: Time spent bootstrapping tablet: real 0.072s user 0.026s sys 0.000s
I20240501 14:09:46.128521 2957 raft_consensus.cc:348] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:46.128630 2957 raft_consensus.cc:729] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Initialized, Role: LEARNER
I20240501 14:09:46.128768 2957 consensus_queue.cc:260] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:46.129050 2957 ts_tablet_manager.cc:1398] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.129954 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session 9438789dc61f4501ad1d4ceaeccee458-96f5e4e25a674a99a6762757f0ab1545 received from {username='slave'} at 127.23.191.70:36823
I20240501 14:09:46.130039 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session 9438789dc61f4501ad1d4ceaeccee458-96f5e4e25a674a99a6762757f0ab1545 on tablet 96f5e4e25a674a99a6762757f0ab1545 with peer 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:46.134068 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:46.141240 31581 consensus_queue.cc:237] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:46.143074 31450 raft_consensus.cc:1261] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:46.143110 31516 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 043344ce36a74a18a0d67ff5bd473e68. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.143482 2891 consensus_queue.cc:1035] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:46.143744 31189 raft_consensus.cc:1261] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:46.144472 2891 consensus_queue.cc:1035] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:46.149718 2891 raft_consensus.cc:2932] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:46.152377 31039 catalog_manager.cc:5484] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.154532 31450 raft_consensus.cc:2932] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:46.166133 31189 raft_consensus.cc:2932] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:46.218395 31581 consensus_queue.cc:237] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } }
I20240501 14:09:46.219945 474 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:46.219978 602 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 14:09:46.220049 31515 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): Couldn't send request to peer 0e4622d80cc94f12b492bdd7597a61e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d2c173b2c32494eb16e46a51ca36e31. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.220407 2683 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:46.220757 1589 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:46.229856 1589 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
I20240501 14:09:46.233176 31039 catalog_manager.cc:5484] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 9 to 10, NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.233856 602 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
I20240501 14:09:46.235972 2405 raft_consensus.cc:1050] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 to VOTER
I20240501 14:09:46.236387 2405 consensus_queue.cc:237] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:46.238915 31450 raft_consensus.cc:1261] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:46.240221 2406 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:46.242283 868 raft_consensus.cc:1261] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:46.242717 2406 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:46.243353 31317 raft_consensus.cc:1261] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:46.244002 2406 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:46.249624 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.249401 474 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
I20240501 14:09:46.253587 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.264389 31298 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.264601 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.264873 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.272073 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.304347 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:46.317498 2406 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:46.317911 31317 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:46.320062 31450 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:46.320488 31046 catalog_manager.cc:5484] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.322099 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:46.328742 31450 consensus_queue.cc:237] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
W20240501 14:09:46.330566 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a542c07a3f841b0a31051a8a37083bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.330660 868 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:46.330931 865 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.331357 382 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.336212 31317 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.336623 382 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.345793 31298 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.348366 31189 consensus_queue.cc:237] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:46.349839 31318 raft_consensus.cc:1261] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:46.350225 865 raft_consensus.cc:1261] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:46.350262 2404 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:46.351215 2404 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:46.352008 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.352455 2404 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:46.358595 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.358989 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6f8bd03c05d84fdaaf138a6d0608e30e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:46.359935 31318 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:46.365370 31039 catalog_manager.cc:5484] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 5 to 6, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.367440 865 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:46.376639 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.377228 31429 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8bd03c05d84fdaaf138a6d0608e30e with delete_type TABLET_DATA_TOMBSTONED (TS 6a6fcc099d1340c383e5a429643d5b7c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41412
I20240501 14:09:46.377414 2456 tablet_replica.cc:330] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: stopping tablet replica
I20240501 14:09:46.377524 2456 raft_consensus.cc:2229] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.377668 2456 raft_consensus.cc:2258] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.377774 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.381431 2456 ts_tablet_manager.cc:1875] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:46.383677 2456 ts_tablet_manager.cc:1888] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:46.383739 2456 log.cc:1196] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/6f8bd03c05d84fdaaf138a6d0608e30e
I20240501 14:09:46.385082 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:46.389626 31032 catalog_manager.cc:4830] TS 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): tablet 6f8bd03c05d84fdaaf138a6d0608e30e (table rebalance_test_table_8 [id=b01dc7a764a349c9a4a9180993e4fbc3]) successfully deleted
I20240501 14:09:46.392518 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:46.395279 382 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:46.398262 31039 catalog_manager.cc:5484] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 6 to 7, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.398257 31317 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:46.405822 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:46.420197 31298 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.424226 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.430773 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.431977 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.432991 2556 raft_consensus.cc:1050] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER a0caf2974041408bb32362bfc4284eaa to VOTER
I20240501 14:09:46.433418 2556 consensus_queue.cc:237] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:46.434638 737 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:46.435007 2513 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:46.436604 31581 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:46.437160 31450 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:46.437450 1555 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:46.438593 1555 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:46.441299 602 raft_consensus.cc:1203] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:09:46.444141 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.446545 868 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:09:46.454983 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.465903 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:46.473095 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.476722 970 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:46.478919 737 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:46.487046 31039 catalog_manager.cc:5484] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:46.487419 31581 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:46.490283 868 raft_consensus.cc:1203] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:09:46.494639 31450 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:46.510953 31450 consensus_queue.cc:237] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:46.513150 868 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:46.513502 31317 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 14:09:46.514039 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a542c07a3f841b0a31051a8a37083bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.514061 382 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:46.514328 382 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:46.514575 3172 ts_tablet_manager.cc:902] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:46.515043 3172 tablet_copy_client.cc:308] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:46.515630 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet 3c8200d9c20e4140906c71c746badb94 from peer b934fd09590a40ada094b9715d990a00 ({username='slave'} at 127.23.191.69:54165)
I20240501 14:09:46.515733 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet 3c8200d9c20e4140906c71c746badb94 from peer b934fd09590a40ada094b9715d990a00 at {username='slave'} at 127.23.191.69:54165: session id = b934fd09590a40ada094b9715d990a00-3c8200d9c20e4140906c71c746badb94
I20240501 14:09:46.516541 31340 tablet_copy_source_session.cc:205] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:46.517453 3172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8200d9c20e4140906c71c746badb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.517874 2175 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:46.520049 3172 tablet_copy_client.cc:783] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:46.520186 3172 tablet_copy_client.cc:647] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:46.521135 31042 catalog_manager.cc:5484] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 7 to 8, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.521193 31318 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:46.524398 868 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:46.526394 31318 consensus_queue.cc:237] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
W20240501 14:09:46.529403 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f26d16499ab54a3e9059f6e4ff041177. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.530179 474 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.530706 970 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.532331 31581 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.533154 970 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.534519 970 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:46.535147 31318 consensus_queue.cc:237] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } }
I20240501 14:09:46.537966 474 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:46.537943 31038 catalog_manager.cc:5484] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 6 to 7, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.542002 3172 tablet_copy_client.cc:523] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:46.543429 3172 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00: Bootstrap starting.
W20240501 14:09:46.543470 31253 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): Couldn't send request to peer 0e4622d80cc94f12b492bdd7597a61e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea66fe3d0da04d18b961927a9c988a20. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.544013 737 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:46.544507 2513 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:46.546869 31580 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:46.547292 2513 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:46.549877 2513 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
I20240501 14:09:46.552110 31580 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
W20240501 14:09:46.553790 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a542c07a3f841b0a31051a8a37083bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.556103 31041 catalog_manager.cc:5484] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 5 to 6, NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
I20240501 14:09:46.556327 31450 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:46.556728 1555 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:46.557860 31450 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
I20240501 14:09:46.565284 737 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
I20240501 14:09:46.571106 31581 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:46.571251 31318 consensus_queue.cc:237] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } }
W20240501 14:09:46.573014 31253 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): Couldn't send request to peer 0e4622d80cc94f12b492bdd7597a61e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea66fe3d0da04d18b961927a9c988a20. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.573684 737 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.575160 3172 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:46.575525 3172 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00: Bootstrap complete.
I20240501 14:09:46.575896 3172 ts_tablet_manager.cc:1367] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00: Time spent bootstrapping tablet: real 0.033s user 0.027s sys 0.000s
I20240501 14:09:46.576454 3172 raft_consensus.cc:348] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:46.576601 3172 raft_consensus.cc:729] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b934fd09590a40ada094b9715d990a00, State: Initialized, Role: LEARNER
I20240501 14:09:46.576705 3172 consensus_queue.cc:260] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:46.577105 3172 ts_tablet_manager.cc:1398] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:46.577574 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session b934fd09590a40ada094b9715d990a00-3c8200d9c20e4140906c71c746badb94 received from {username='slave'} at 127.23.191.69:54165
I20240501 14:09:46.577664 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session b934fd09590a40ada094b9715d990a00-3c8200d9c20e4140906c71c746badb94 on tablet 3c8200d9c20e4140906c71c746badb94 with peer b934fd09590a40ada094b9715d990a00
I20240501 14:09:46.578281 2556 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.578444 3204 ts_tablet_manager.cc:902] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:46.578922 3204 tablet_copy_client.cc:308] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:46.579375 31448 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.579806 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet a7810938b8fb406e9c6ac438f54afd03 from peer a63e0dd1ba2e4a8db8ace9718a700c32 ({username='slave'} at 127.23.191.72:59731)
I20240501 14:09:46.579922 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet a7810938b8fb406e9c6ac438f54afd03 from peer a63e0dd1ba2e4a8db8ace9718a700c32 at {username='slave'} at 127.23.191.72:59731: session id = a63e0dd1ba2e4a8db8ace9718a700c32-a7810938b8fb406e9c6ac438f54afd03
I20240501 14:09:46.580351 2513 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.581037 31209 tablet_copy_source_session.cc:205] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:46.582310 3204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7810938b8fb406e9c6ac438f54afd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.582182 3210 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
I20240501 14:09:46.583067 737 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
I20240501 14:09:46.584144 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ea66fe3d0da04d18b961927a9c988a20 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:46.584753 3204 tablet_copy_client.cc:783] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:46.584923 3204 tablet_copy_client.cc:647] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:46.585024 31450 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
I20240501 14:09:46.585559 31042 catalog_manager.cc:5484] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 6 to 7, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.586782 3204 tablet_copy_client.cc:523] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:46.588321 3204 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap starting.
I20240501 14:09:46.589450 31561 tablet_service.cc:1514] Processing DeleteTablet for tablet ea66fe3d0da04d18b961927a9c988a20 with delete_type TABLET_DATA_TOMBSTONED (TS e7eada5032fb45c3a5e3196e2b3faaab not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34312
I20240501 14:09:46.591030 3227 ts_tablet_manager.cc:902] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:46.591521 3227 tablet_copy_client.cc:308] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
W20240501 14:09:46.592279 31361 debug-util.cc:398] Leaking SignalData structure 0x5649b514c5e0 after lost signal to thread 2449
I20240501 14:09:46.598774 31581 consensus_queue.cc:237] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } }
W20240501 14:09:46.601138 31518 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265): Couldn't send request to peer 30e0b8849e3841ca9ab0de01e719ee4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ad9c714b2b4fff94063d2838835173. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.601814 31450 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240501 14:09:46.602078 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ad9c714b2b4fff94063d2838835173. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.602416 2877 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:46.603201 474 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:46.603385 3248 ts_tablet_manager.cc:902] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:46.603763 2446 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:46.603993 3248 tablet_copy_client.cc:308] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:46.605127 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.606036 3211 ts_tablet_manager.cc:902] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:46.606523 31318 consensus_queue.cc:237] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } }
I20240501 14:09:46.606834 3211 tablet_copy_client.cc:308] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:46.606853 2446 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } } }
I20240501 14:09:46.607331 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet 5b2e30cb9de14fe19273b3cef801cb67 from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:42437)
I20240501 14:09:46.607442 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet 5b2e30cb9de14fe19273b3cef801cb67 from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:42437: session id = a0caf2974041408bb32362bfc4284eaa-5b2e30cb9de14fe19273b3cef801cb67
I20240501 14:09:46.608224 602 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:46.608227 31448 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } } }
I20240501 14:09:46.608429 31209 tablet_copy_source_session.cc:205] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:46.609143 3210 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:46.609596 474 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:46.609889 472 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } } }
I20240501 14:09:46.610469 31038 catalog_manager.cc:5484] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 11 to 12, NON_VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:46.612006 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): Couldn't send request to peer e7eada5032fb45c3a5e3196e2b3faaab. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.612483 3210 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:46.613045 3211 ts_tablet_manager.cc:1875] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:09:46.613903 31189 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:46.614424 970 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:46.614552 2556 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.615319 602 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.617769 31042 catalog_manager.cc:5484] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 reported cstate change: config changed from index 7 to 8, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.617910 31189 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.624853 31601 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet 72ad9c714b2b4fff94063d2838835173 from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:41541)
I20240501 14:09:46.624966 31601 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet 72ad9c714b2b4fff94063d2838835173 from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:41541: session id = a0caf2974041408bb32362bfc4284eaa-72ad9c714b2b4fff94063d2838835173
I20240501 14:09:46.625372 3210 raft_consensus.cc:1050] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER b934fd09590a40ada094b9715d990a00 to VOTER
I20240501 14:09:46.625890 31601 tablet_copy_source_session.cc:205] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:46.625816 3210 consensus_queue.cc:237] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } }
I20240501 14:09:46.626222 31602 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet 043344ce36a74a18a0d67ff5bd473e68 from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:41541)
I20240501 14:09:46.626302 31602 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet 043344ce36a74a18a0d67ff5bd473e68 from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:41541: session id = a0caf2974041408bb32362bfc4284eaa-043344ce36a74a18a0d67ff5bd473e68
I20240501 14:09:46.627130 31602 tablet_copy_source_session.cc:205] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:46.627476 472 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
W20240501 14:09:46.628283 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): Couldn't send request to peer e7eada5032fb45c3a5e3196e2b3faaab. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.628722 31189 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:46.629570 2556 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:46.630098 3242 ts_tablet_manager.cc:902] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:46.630244 602 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:46.630769 3242 tablet_copy_client.cc:308] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:46.630848 474 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:46.630968 2556 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:46.631333 31602 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet 2e68be1521964ba8b864801fdf3b0df3 from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:41541)
I20240501 14:09:46.631333 1555 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:46.631420 31602 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet 2e68be1521964ba8b864801fdf3b0df3 from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:41541: session id = a0caf2974041408bb32362bfc4284eaa-2e68be1521964ba8b864801fdf3b0df3
I20240501 14:09:46.632184 31602 tablet_copy_source_session.cc:205] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:46.632922 2556 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.635171 31189 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.636735 602 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.638599 3248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043344ce36a74a18a0d67ff5bd473e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.639554 3211 ts_tablet_manager.cc:1888] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240501 14:09:46.639727 3211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2e30cb9de14fe19273b3cef801cb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.640590 3248 tablet_copy_client.cc:783] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:46.640748 3248 tablet_copy_client.cc:647] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:46.640590 3211 tablet_copy_client.cc:783] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:46.640933 3242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e68be1521964ba8b864801fdf3b0df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.641335 3227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ad9c714b2b4fff94063d2838835173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.641616 31042 catalog_manager.cc:5484] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 8 to 9, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.642079 3248 tablet_copy_client.cc:523] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:46.643020 3242 tablet_copy_client.cc:783] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:46.643210 3242 tablet_copy_client.cc:647] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:46.643323 3248 tablet_bootstrap.cc:492] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:46.645037 3242 tablet_copy_client.cc:523] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:46.646409 3242 tablet_bootstrap.cc:492] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:46.643376 3227 tablet_copy_client.cc:783] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:46.648310 3227 tablet_copy_client.cc:647] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:46.650245 3227 tablet_copy_client.cc:523] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:46.650210 474 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.651228 3227 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:46.653672 31318 consensus_queue.cc:237] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:46.655403 602 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:46.655833 1555 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:46.656870 31189 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:46.657708 474 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:46.658080 1555 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:46.658655 1555 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:46.660567 1555 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.663952 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3c8200d9c20e4140906c71c746badb94 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:46.664323 31040 catalog_manager.cc:5484] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 9 to 10, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.664462 31189 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.664462 602 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.667574 31561 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c8200d9c20e4140906c71c746badb94 with delete_type TABLET_DATA_TOMBSTONED (TS e7eada5032fb45c3a5e3196e2b3faaab not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:34312
I20240501 14:09:46.669098 3226 tablet_replica.cc:330] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab: stopping tablet replica
I20240501 14:09:46.669375 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.669445 3226 raft_consensus.cc:2229] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.670466 3226 raft_consensus.cc:2258] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.670964 474 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.671267 3226 ts_tablet_manager.cc:1875] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:46.672103 31318 consensus_queue.cc:237] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:46.673704 3226 ts_tablet_manager.cc:1888] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:09:46.673825 3226 log.cc:1196] T ea66fe3d0da04d18b961927a9c988a20 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/wals/ea66fe3d0da04d18b961927a9c988a20
I20240501 14:09:46.674743 31033 catalog_manager.cc:4830] TS e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): tablet ea66fe3d0da04d18b961927a9c988a20 (table rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]) successfully deleted
I20240501 14:09:46.675014 31033 catalog_manager.cc:4830] TS e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): tablet 3c8200d9c20e4140906c71c746badb94 (table rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]) successfully deleted
I20240501 14:09:46.640923 3211 tablet_copy_client.cc:647] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:46.675585 602 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:46.675967 2513 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:46.676596 3211 tablet_copy_client.cc:523] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:46.677913 472 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:46.678428 3258 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:46.681249 382 raft_consensus.cc:1050] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c: attempting to promote NON_VOTER b934fd09590a40ada094b9715d990a00 to VOTER
I20240501 14:09:46.681632 382 consensus_queue.cc:237] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:46.683928 472 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:46.683926 31581 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:46.684480 31189 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:46.684446 2175 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:46.684675 2419 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:46.685285 2419 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:46.686259 970 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.686638 603 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.688478 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3c8200d9c20e4140906c71c746badb94 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:46.689033 474 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.689767 31042 catalog_manager.cc:5484] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 10 to 11, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:46.691710 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:46.693209 3211 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:46.693570 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c8200d9c20e4140906c71c746badb94 with delete_type TABLET_DATA_TOMBSTONED (TS 0e4622d80cc94f12b492bdd7597a61e1 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:59614
I20240501 14:09:46.695961 3288 tablet_replica.cc:330] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: stopping tablet replica
I20240501 14:09:46.696130 3288 raft_consensus.cc:2229] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.696388 3288 raft_consensus.cc:2258] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.697706 3288 ts_tablet_manager.cc:1875] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:46.700986 31318 consensus_queue.cc:237] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:46.702175 3288 ts_tablet_manager.cc:1888] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 14:09:46.702983 3288 log.cc:1196] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/3c8200d9c20e4140906c71c746badb94
I20240501 14:09:46.703629 31031 catalog_manager.cc:4830] TS 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): tablet 3c8200d9c20e4140906c71c746badb94 (table rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]) successfully deleted
I20240501 14:09:46.704496 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.705082 734 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:09:46.706117 31256 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4603d1b1b1d441b1b0be0464e2f2c9e2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.706301 2513 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.707326 31298 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.708626 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:46.709244 31450 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.709617 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.709723 3210 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.710059 3281 ts_tablet_manager.cc:902] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:46.710889 3281 tablet_copy_client.cc:308] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:46.714571 31602 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet 9d2c173b2c32494eb16e46a51ca36e31 from peer 0e4622d80cc94f12b492bdd7597a61e1 ({username='slave'} at 127.23.191.65:33587)
I20240501 14:09:46.714666 31602 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet 9d2c173b2c32494eb16e46a51ca36e31 from peer 0e4622d80cc94f12b492bdd7597a61e1 at {username='slave'} at 127.23.191.65:33587: session id = 0e4622d80cc94f12b492bdd7597a61e1-9d2c173b2c32494eb16e46a51ca36e31
I20240501 14:09:46.715567 31602 tablet_copy_source_session.cc:205] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:46.728971 3281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2c173b2c32494eb16e46a51ca36e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.734196 3284 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.735064 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.738318 31040 catalog_manager.cc:5484] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.740626 31581 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.744345 31189 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.749265 3281 tablet_copy_client.cc:783] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:46.749495 3281 tablet_copy_client.cc:647] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:46.750942 3281 tablet_copy_client.cc:523] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:46.752774 3281 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:46.753731 3210 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:46.756541 31042 catalog_manager.cc:5484] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.757069 31450 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:46.760861 734 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:46.764034 472 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.767920 31450 consensus_queue.cc:237] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
W20240501 14:09:46.770601 31387 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ffdb76b2c5149a18d45bee9581e605a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.771356 31448 consensus_queue.cc:237] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } }
I20240501 14:09:46.772534 472 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:46.772699 734 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:09:46.772920 31032 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 8ffdb76b2c5149a18d45bee9581e605a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:46.773087 2175 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.773311 31188 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.773814 3284 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.775738 31189 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:46.776175 31581 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:46.776501 2175 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:46.776859 2175 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:09:46.777328 31386 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): Couldn't send request to peer e7eada5032fb45c3a5e3196e2b3faaab. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.777675 2175 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:46.781555 3284 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:46.783810 31189 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:46.785113 3242 tablet_bootstrap.cc:492] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:46.785296 31038 catalog_manager.cc:5484] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 5 to 6, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.785490 3242 tablet_bootstrap.cc:492] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:46.785835 3242 ts_tablet_manager.cc:1367] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.139s user 0.023s sys 0.004s
I20240501 14:09:46.786589 472 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:46.787715 3242 raft_consensus.cc:348] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:46.787827 3242 raft_consensus.cc:729] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:46.787915 3242 consensus_queue.cc:260] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:46.788173 3242 ts_tablet_manager.cc:1398] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:09:46.789073 31602 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-2e68be1521964ba8b864801fdf3b0df3 received from {username='slave'} at 127.23.191.71:41541
I20240501 14:09:46.789155 31602 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-2e68be1521964ba8b864801fdf3b0df3 on tablet 2e68be1521964ba8b864801fdf3b0df3 with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:46.795061 31448 consensus_queue.cc:237] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:46.796450 31189 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.796896 382 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.797467 3204 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:46.797793 3204 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32: Bootstrap complete.
I20240501 14:09:46.797976 3204 ts_tablet_manager.cc:1367] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent bootstrapping tablet: real 0.210s user 0.028s sys 0.001s
I20240501 14:09:46.798425 3204 raft_consensus.cc:348] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:46.798552 3204 raft_consensus.cc:729] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63e0dd1ba2e4a8db8ace9718a700c32, State: Initialized, Role: LEARNER
I20240501 14:09:46.798652 3204 consensus_queue.cc:260] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:46.798941 3204 ts_tablet_manager.cc:1398] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:46.799388 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-a7810938b8fb406e9c6ac438f54afd03 received from {username='slave'} at 127.23.191.72:59731
I20240501 14:09:46.799459 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session a63e0dd1ba2e4a8db8ace9718a700c32-a7810938b8fb406e9c6ac438f54afd03 on tablet a7810938b8fb406e9c6ac438f54afd03 with peer a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:46.803345 474 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.803754 382 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.804967 31580 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.805716 382 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:46.806522 31581 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:46.808956 382 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.811103 31189 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.813582 31032 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ffdb76b2c5149a18d45bee9581e605a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:46.814275 31040 catalog_manager.cc:5484] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 6 to 7, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.816012 472 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.816722 31581 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.817179 31032 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 8ffdb76b2c5149a18d45bee9581e605a with cas_config_opid_index 5: aborting the task: latest config opid_index 7; task opid_index 5
I20240501 14:09:46.817797 586 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ffdb76b2c5149a18d45bee9581e605a with delete_type TABLET_DATA_TOMBSTONED (TS 9438789dc61f4501ad1d4ceaeccee458 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46552
I20240501 14:09:46.821790 31448 consensus_queue.cc:237] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:46.823055 3281 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240501 14:09:46.823554 31034 catalog_manager.cc:4797] TS 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): delete failed for tablet 8ffdb76b2c5149a18d45bee9581e605a because the tablet was not found. No further retry: Not found: Tablet not found: 8ffdb76b2c5149a18d45bee9581e605a
I20240501 14:09:46.823060 31189 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:46.823953 3281 tablet_bootstrap.cc:492] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap complete.
I20240501 14:09:46.824030 2175 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:46.824151 3281 ts_tablet_manager.cc:1367] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.071s user 0.031s sys 0.000s
I20240501 14:09:46.824548 3281 raft_consensus.cc:348] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } }
I20240501 14:09:46.824720 3281 raft_consensus.cc:729] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: LEARNER
I20240501 14:09:46.824882 3281 consensus_queue.cc:260] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } }
I20240501 14:09:46.825841 2175 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.826835 3281 ts_tablet_manager.cc:1398] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:46.828109 31032 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ffdb76b2c5149a18d45bee9581e605a with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:46.828403 472 raft_consensus.cc:1261] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:46.828907 31602 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session 0e4622d80cc94f12b492bdd7597a61e1-9d2c173b2c32494eb16e46a51ca36e31 received from {username='slave'} at 127.23.191.65:33587
I20240501 14:09:46.828979 31602 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session 0e4622d80cc94f12b492bdd7597a61e1-9d2c173b2c32494eb16e46a51ca36e31 on tablet 9d2c173b2c32494eb16e46a51ca36e31 with peer 0e4622d80cc94f12b492bdd7597a61e1
I20240501 14:09:46.830519 31189 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.833032 382 consensus_queue.cc:1035] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:46.833050 31038 catalog_manager.cc:5484] T 8ffdb76b2c5149a18d45bee9581e605a P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 7 to 8, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.834328 472 raft_consensus.cc:2932] T 8ffdb76b2c5149a18d45bee9581e605a P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:46.835435 31561 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ffdb76b2c5149a18d45bee9581e605a with delete_type TABLET_DATA_TOMBSTONED (TS e7eada5032fb45c3a5e3196e2b3faaab not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34312
I20240501 14:09:46.836644 3226 tablet_replica.cc:330] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab: stopping tablet replica
I20240501 14:09:46.836905 3226 raft_consensus.cc:2229] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.837170 3226 raft_consensus.cc:2258] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.838016 3226 ts_tablet_manager.cc:1875] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:46.840086 3226 ts_tablet_manager.cc:1888] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:09:46.840211 3226 log.cc:1196] T 8ffdb76b2c5149a18d45bee9581e605a P e7eada5032fb45c3a5e3196e2b3faaab: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/wals/8ffdb76b2c5149a18d45bee9581e605a
I20240501 14:09:46.840770 31033 catalog_manager.cc:4830] TS e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): tablet 8ffdb76b2c5149a18d45bee9581e605a (table rebalance_test_table_3 [id=351ff5b1095247928fbb77893616e4a9]) successfully deleted
I20240501 14:09:46.850836 3284 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.851369 31188 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.856575 31038 catalog_manager.cc:5484] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 6 to 7, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.856673 734 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.868495 3227 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:46.868975 3227 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:46.869484 3227 ts_tablet_manager.cc:1367] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.218s user 0.030s sys 0.000s
I20240501 14:09:46.872028 3227 raft_consensus.cc:348] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } }
I20240501 14:09:46.873836 3227 raft_consensus.cc:729] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:46.874047 3227 consensus_queue.cc:260] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } }
I20240501 14:09:46.875550 3258 raft_consensus.cc:1050] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 to VOTER
I20240501 14:09:46.876025 3258 consensus_queue.cc:237] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:46.877732 31188 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:46.878150 603 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:46.878199 3227 ts_tablet_manager.cc:1398] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:46.877602 3248 tablet_bootstrap.cc:492] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:46.878628 3248 tablet_bootstrap.cc:492] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:46.878765 31602 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-72ad9c714b2b4fff94063d2838835173 received from {username='slave'} at 127.23.191.71:41541
I20240501 14:09:46.878809 3248 ts_tablet_manager.cc:1367] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.236s user 0.024s sys 0.000s
I20240501 14:09:46.878832 31602 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-72ad9c714b2b4fff94063d2838835173 on tablet 72ad9c714b2b4fff94063d2838835173 with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:46.879004 1555 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:46.879235 3248 raft_consensus.cc:348] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:46.879339 3248 raft_consensus.cc:729] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:46.879434 3248 consensus_queue.cc:260] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:46.879673 3248 ts_tablet_manager.cc:1398] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:46.880051 31602 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-043344ce36a74a18a0d67ff5bd473e68 received from {username='slave'} at 127.23.191.71:41541
I20240501 14:09:46.880115 31602 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-043344ce36a74a18a0d67ff5bd473e68 on tablet 043344ce36a74a18a0d67ff5bd473e68 with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:46.880481 865 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:46.882524 1555 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:46.888240 1555 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:46.892112 1555 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:46.892577 31581 consensus_queue.cc:237] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:46.894227 603 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:46.894448 472 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.894433 31188 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:46.894999 2683 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.895807 31448 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.896734 31040 catalog_manager.cc:5484] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 7 to 8, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:46.896950 31518 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b83f4cd48b4b4251a8ac4ddebdb8e4e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.898187 2683 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.904340 2683 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:46.905748 31318 consensus_queue.cc:237] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } }
I20240501 14:09:46.906847 31448 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:46.907822 31038 catalog_manager.cc:5484] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.908260 31188 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 14:09:46.908526 31255 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): Couldn't send request to peer e7eada5032fb45c3a5e3196e2b3faaab. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.908845 970 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:46.910809 865 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:46.911377 472 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:46.916860 2513 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:46.917819 865 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:46.920725 603 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:46.921427 2513 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:46.923298 3258 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.925397 603 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.928568 31188 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.928851 3211 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:46.929155 3211 tablet_bootstrap.cc:492] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:46.929384 3211 ts_tablet_manager.cc:1367] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.236s user 0.023s sys 0.004s
I20240501 14:09:46.929831 3211 raft_consensus.cc:348] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:46.930001 3211 raft_consensus.cc:729] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:46.930104 3211 consensus_queue.cc:260] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:09:46.930444 3211 ts_tablet_manager.cc:1398] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.931039 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-5b2e30cb9de14fe19273b3cef801cb67 received from {username='slave'} at 127.23.191.71:42437
I20240501 14:09:46.931110 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-5b2e30cb9de14fe19273b3cef801cb67 on tablet 5b2e30cb9de14fe19273b3cef801cb67 with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:46.932045 31043 catalog_manager.cc:5484] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 8 to 9, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.934651 865 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.948657 31318 consensus_queue.cc:237] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } }
I20240501 14:09:46.950893 865 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:46.951272 603 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:46.952006 2513 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:46.953024 2513 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:09:46.953202 31255 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): Couldn't send request to peer e7eada5032fb45c3a5e3196e2b3faaab. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.954629 2513 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.954977 865 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.956357 603 raft_consensus.cc:2932] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.959834 31042 catalog_manager.cc:5484] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 9438789dc61f4501ad1d4ceaeccee458 reported cstate change: config changed from index 9 to 10, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:09:46.960606 3281 ts_tablet_manager.cc:902] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:46.960814 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f91909d3b1b46e9b858ce5c5b2e7975 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:46.961058 3281 tablet_copy_client.cc:308] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:46.961853 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet ea66fe3d0da04d18b961927a9c988a20 from peer 0e4622d80cc94f12b492bdd7597a61e1 ({username='slave'} at 127.23.191.65:46557)
I20240501 14:09:46.961951 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet ea66fe3d0da04d18b961927a9c988a20 from peer 0e4622d80cc94f12b492bdd7597a61e1 at {username='slave'} at 127.23.191.65:46557: session id = 0e4622d80cc94f12b492bdd7597a61e1-ea66fe3d0da04d18b961927a9c988a20
I20240501 14:09:46.962909 31340 tablet_copy_source_session.cc:205] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:46.963881 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f91909d3b1b46e9b858ce5c5b2e7975 with delete_type TABLET_DATA_TOMBSTONED (TS 0e4622d80cc94f12b492bdd7597a61e1 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59614
I20240501 14:09:46.964077 3288 tablet_replica.cc:330] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1: stopping tablet replica
I20240501 14:09:46.964216 3288 raft_consensus.cc:2229] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.964457 3288 raft_consensus.cc:2258] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.966055 3288 ts_tablet_manager.cc:1875] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:46.968003 3288 ts_tablet_manager.cc:1888] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:09:46.968066 3288 log.cc:1196] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/8f91909d3b1b46e9b858ce5c5b2e7975
I20240501 14:09:46.968602 31031 catalog_manager.cc:4830] TS 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): tablet 8f91909d3b1b46e9b858ce5c5b2e7975 (table rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]) successfully deleted
I20240501 14:09:46.969137 3281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea66fe3d0da04d18b961927a9c988a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.970992 3281 tablet_copy_client.cc:783] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:46.971118 3281 tablet_copy_client.cc:647] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:46.973254 3281 tablet_copy_client.cc:523] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:46.974400 3281 tablet_bootstrap.cc:492] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
I20240501 14:09:46.984925 31188 consensus_queue.cc:237] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } }
I20240501 14:09:46.987321 865 raft_consensus.cc:1261] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:46.987393 31318 raft_consensus.cc:1261] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:09:46.987748 31123 consensus_peers.cc:485] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): Couldn't send request to peer 6a6fcc099d1340c383e5a429643d5b7c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:46.988094 2405 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.988405 2405 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:46.990356 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.991425 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:46.991930 2405 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } } }
I20240501 14:09:46.995126 31046 catalog_manager.cc:5484] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 6 to 7, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.997493 865 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } } }
I20240501 14:09:47.003038 31318 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } } }
I20240501 14:09:47.004556 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.007521 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:47.008127 865 raft_consensus.cc:1203] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 14:09:47.016992 3433 ts_tablet_manager.cc:902] T f26d16499ab54a3e9059f6e4ff041177 P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:47.017215 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:47.017585 3433 tablet_copy_client.cc:308] T f26d16499ab54a3e9059f6e4ff041177 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:47.018266 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet f26d16499ab54a3e9059f6e4ff041177 from peer 9438789dc61f4501ad1d4ceaeccee458 ({username='slave'} at 127.23.191.70:41145)
I20240501 14:09:47.018373 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet f26d16499ab54a3e9059f6e4ff041177 from peer 9438789dc61f4501ad1d4ceaeccee458 at {username='slave'} at 127.23.191.70:41145: session id = 9438789dc61f4501ad1d4ceaeccee458-f26d16499ab54a3e9059f6e4ff041177
I20240501 14:09:47.019270 31340 tablet_copy_source_session.cc:205] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:47.020257 3433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26d16499ab54a3e9059f6e4ff041177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.022761 3433 tablet_copy_client.cc:783] T f26d16499ab54a3e9059f6e4ff041177 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:47.022958 3433 tablet_copy_client.cc:647] T f26d16499ab54a3e9059f6e4ff041177 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:47.024127 3433 tablet_copy_client.cc:523] T f26d16499ab54a3e9059f6e4ff041177 P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:47.026185 734 raft_consensus.cc:1203] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20240501 14:09:47.026934 3419 raft_consensus.cc:1050] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 to VOTER
I20240501 14:09:47.027339 3419 consensus_queue.cc:237] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:47.028530 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.029659 472 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:47.029661 602 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:47.030223 3419 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:47.030535 31581 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:47.030563 3419 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:47.031775 3419 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:47.032522 3433 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap starting.
I20240501 14:09:47.033672 31298 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:47.036751 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.043591 31188 raft_consensus.cc:1203] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 14:09:47.045770 734 raft_consensus.cc:1203] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:09:47.056301 734 raft_consensus.cc:1203] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:47.076493 31188 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } }
I20240501 14:09:47.076740 3211 tablet_replica.cc:330] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: stopping tablet replica
I20240501 14:09:47.076898 3211 raft_consensus.cc:2229] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:09:47.077000 3211 raft_consensus.cc:2258] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:09:47.077828 3211 ts_tablet_manager.cc:1875] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:47.078572 603 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
W20240501 14:09:47.079003 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.079180 3439 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 14:09:47.079993 3211 ts_tablet_manager.cc:1888] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 14:09:47.080056 3211 log.cc:1196] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-6/wal/wals/72ad9c714b2b4fff94063d2838835173
I20240501 14:09:47.080312 3211 ts_tablet_manager.cc:902] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:47.080667 3211 tablet_copy_client.cc:308] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:47.081604 734 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 14:09:47.082056 3439 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 14:09:47.082278 31602 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet 72ad9c714b2b4fff94063d2838835173 from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:41541)
I20240501 14:09:47.082384 31602 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet 72ad9c714b2b4fff94063d2838835173 from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:41541: session id = a0caf2974041408bb32362bfc4284eaa-72ad9c714b2b4fff94063d2838835173
I20240501 14:09:47.083333 31602 tablet_copy_source_session.cc:205] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:47.084033 3449 ts_tablet_manager.cc:902] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:47.085338 3439 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:47.086166 3281 tablet_bootstrap.cc:492] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:47.086417 3281 tablet_bootstrap.cc:492] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap complete.
I20240501 14:09:47.087656 3281 ts_tablet_manager.cc:1367] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.113s user 0.025s sys 0.001s
I20240501 14:09:47.087777 602 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:47.088073 3281 raft_consensus.cc:348] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } }
I20240501 14:09:47.088161 3281 raft_consensus.cc:729] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: LEARNER
I20240501 14:09:47.088254 3281 consensus_queue.cc:260] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } }
I20240501 14:09:47.088383 31038 catalog_manager.cc:5484] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 7 to 8, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:47.088567 3281 ts_tablet_manager.cc:1398] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.089012 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session 0e4622d80cc94f12b492bdd7597a61e1-ea66fe3d0da04d18b961927a9c988a20 received from {username='slave'} at 127.23.191.65:46557
I20240501 14:09:47.089084 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session 0e4622d80cc94f12b492bdd7597a61e1-ea66fe3d0da04d18b961927a9c988a20 on tablet ea66fe3d0da04d18b961927a9c988a20 with peer 0e4622d80cc94f12b492bdd7597a61e1
I20240501 14:09:47.089653 31581 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:47.090731 3449 tablet_copy_client.cc:308] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:47.091265 3450 ts_tablet_manager.cc:902] T 7a542c07a3f841b0a31051a8a37083bf P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:47.091521 31602 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet 72ad9c714b2b4fff94063d2838835173 from peer 30e0b8849e3841ca9ab0de01e719ee4a ({username='slave'} at 127.23.191.66:43501)
I20240501 14:09:47.091625 31602 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet 72ad9c714b2b4fff94063d2838835173 from peer 30e0b8849e3841ca9ab0de01e719ee4a at {username='slave'} at 127.23.191.66:43501: session id = 30e0b8849e3841ca9ab0de01e719ee4a-72ad9c714b2b4fff94063d2838835173
I20240501 14:09:47.091899 3450 tablet_copy_client.cc:308] T 7a542c07a3f841b0a31051a8a37083bf P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:39413
I20240501 14:09:47.092440 31602 tablet_copy_source_session.cc:205] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:47.094087 2405 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:47.095744 603 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:47.095746 472 raft_consensus.cc:2932] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:09:47.096611 3433 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:47.096684 734 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:47.096941 3433 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap complete.
I20240501 14:09:47.097097 3433 ts_tablet_manager.cc:1367] T f26d16499ab54a3e9059f6e4ff041177 P 9438789dc61f4501ad1d4ceaeccee458: Time spent bootstrapping tablet: real 0.065s user 0.022s sys 0.005s
I20240501 14:09:47.097488 3433 raft_consensus.cc:348] T f26d16499ab54a3e9059f6e4ff041177 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:47.097764 3433 raft_consensus.cc:729] T f26d16499ab54a3e9059f6e4ff041177 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Initialized, Role: LEARNER
I20240501 14:09:47.097877 3433 consensus_queue.cc:260] T f26d16499ab54a3e9059f6e4ff041177 P 9438789dc61f4501ad1d4ceaeccee458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:47.098228 3433 ts_tablet_manager.cc:1398] T f26d16499ab54a3e9059f6e4ff041177 P 9438789dc61f4501ad1d4ceaeccee458: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:47.098681 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session 9438789dc61f4501ad1d4ceaeccee458-f26d16499ab54a3e9059f6e4ff041177 received from {username='slave'} at 127.23.191.70:41145
I20240501 14:09:47.098752 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session 9438789dc61f4501ad1d4ceaeccee458-f26d16499ab54a3e9059f6e4ff041177 on tablet f26d16499ab54a3e9059f6e4ff041177 with peer 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:47.099139 31047 catalog_manager.cc:5484] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 14 to 15, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true } } }
I20240501 14:09:47.100441 3449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ad9c714b2b4fff94063d2838835173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.102916 3449 tablet_copy_client.cc:783] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:47.103091 3449 tablet_copy_client.cc:647] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:47.103955 3211 ts_tablet_manager.cc:1875] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:09:47.105306 3449 tablet_copy_client.cc:523] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:47.106789 3449 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:09:47.112529 3211 ts_tablet_manager.cc:1888] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.12
I20240501 14:09:47.115021 3211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ad9c714b2b4fff94063d2838835173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.120455 3211 tablet_copy_client.cc:783] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:47.120659 3211 tablet_copy_client.cc:647] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:47.122025 3211 tablet_copy_client.cc:523] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:47.123715 3211 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:47.126305 3248 ts_tablet_manager.cc:902] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:47.126765 3248 tablet_copy_client.cc:308] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:39413
I20240501 14:09:47.127166 31471 tablet_copy_service.cc:140] P 6a6fcc099d1340c383e5a429643d5b7c: Received BeginTabletCopySession request for tablet 7a542c07a3f841b0a31051a8a37083bf from peer a0caf2974041408bb32362bfc4284eaa ({username='slave'} at 127.23.191.71:41729)
I20240501 14:09:47.127259 31471 tablet_copy_service.cc:161] P 6a6fcc099d1340c383e5a429643d5b7c: Beginning new tablet copy session on tablet 7a542c07a3f841b0a31051a8a37083bf from peer a0caf2974041408bb32362bfc4284eaa at {username='slave'} at 127.23.191.71:41729: session id = a0caf2974041408bb32362bfc4284eaa-7a542c07a3f841b0a31051a8a37083bf
I20240501 14:09:47.128120 31471 tablet_copy_source_session.cc:205] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:47.128991 3248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a542c07a3f841b0a31051a8a37083bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.130417 31471 tablet_copy_service.cc:140] P 6a6fcc099d1340c383e5a429643d5b7c: Received BeginTabletCopySession request for tablet 7a542c07a3f841b0a31051a8a37083bf from peer 9438789dc61f4501ad1d4ceaeccee458 ({username='slave'} at 127.23.191.70:52881)
I20240501 14:09:47.130504 31471 tablet_copy_service.cc:161] P 6a6fcc099d1340c383e5a429643d5b7c: Beginning new tablet copy session on tablet 7a542c07a3f841b0a31051a8a37083bf from peer 9438789dc61f4501ad1d4ceaeccee458 at {username='slave'} at 127.23.191.70:52881: session id = 9438789dc61f4501ad1d4ceaeccee458-7a542c07a3f841b0a31051a8a37083bf
I20240501 14:09:47.131093 3248 tablet_copy_client.cc:783] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:47.131229 3248 tablet_copy_client.cc:647] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:47.131300 31471 tablet_copy_source_session.cc:205] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:47.132299 3450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a542c07a3f841b0a31051a8a37083bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.135123 3450 tablet_copy_client.cc:783] T 7a542c07a3f841b0a31051a8a37083bf P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:47.135272 3450 tablet_copy_client.cc:647] T 7a542c07a3f841b0a31051a8a37083bf P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:47.136544 3450 tablet_copy_client.cc:523] T 7a542c07a3f841b0a31051a8a37083bf P 9438789dc61f4501ad1d4ceaeccee458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:47.137838 3450 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap starting.
I20240501 14:09:47.142498 3248 tablet_copy_client.cc:523] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:47.143396 3449 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:47.143685 3449 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap complete.
I20240501 14:09:47.143867 3449 ts_tablet_manager.cc:1367] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.037s user 0.026s sys 0.000s
I20240501 14:09:47.144341 3248 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa: Bootstrap starting.
I20240501 14:09:47.144309 3449 raft_consensus.cc:348] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } }
I20240501 14:09:47.144445 3449 raft_consensus.cc:729] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: LEARNER
I20240501 14:09:47.144544 3449 consensus_queue.cc:260] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } }
I20240501 14:09:47.144852 3449 ts_tablet_manager.cc:1398] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:09:47.145587 31602 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session 30e0b8849e3841ca9ab0de01e719ee4a-72ad9c714b2b4fff94063d2838835173 received from {username='slave'} at 127.23.191.66:43501
I20240501 14:09:47.145696 31602 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session 30e0b8849e3841ca9ab0de01e719ee4a-72ad9c714b2b4fff94063d2838835173 on tablet 72ad9c714b2b4fff94063d2838835173 with peer 30e0b8849e3841ca9ab0de01e719ee4a
I20240501 14:09:47.169762 3524 ts_tablet_manager.cc:902] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Initiating tablet copy from peer 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:09:47.170629 3524 tablet_copy_client.cc:308] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:39413
I20240501 14:09:47.171242 31471 tablet_copy_service.cc:140] P 6a6fcc099d1340c383e5a429643d5b7c: Received BeginTabletCopySession request for tablet ad0dac76fd0c4ae6b8e0549376c98e5d from peer e7eada5032fb45c3a5e3196e2b3faaab ({username='slave'} at 127.23.191.68:50033)
I20240501 14:09:47.171340 31471 tablet_copy_service.cc:161] P 6a6fcc099d1340c383e5a429643d5b7c: Beginning new tablet copy session on tablet ad0dac76fd0c4ae6b8e0549376c98e5d from peer e7eada5032fb45c3a5e3196e2b3faaab at {username='slave'} at 127.23.191.68:50033: session id = e7eada5032fb45c3a5e3196e2b3faaab-ad0dac76fd0c4ae6b8e0549376c98e5d
I20240501 14:09:47.172235 31471 tablet_copy_source_session.cc:205] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:47.174201 3524 ts_tablet_manager.cc:1875] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:09:47.174571 3450 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:47.174878 3450 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P 9438789dc61f4501ad1d4ceaeccee458: Bootstrap complete.
I20240501 14:09:47.175108 3450 ts_tablet_manager.cc:1367] T 7a542c07a3f841b0a31051a8a37083bf P 9438789dc61f4501ad1d4ceaeccee458: Time spent bootstrapping tablet: real 0.037s user 0.021s sys 0.007s
I20240501 14:09:47.175580 3450 raft_consensus.cc:348] T 7a542c07a3f841b0a31051a8a37083bf P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:47.175729 3450 raft_consensus.cc:729] T 7a542c07a3f841b0a31051a8a37083bf P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Initialized, Role: LEARNER
I20240501 14:09:47.175832 3450 consensus_queue.cc:260] T 7a542c07a3f841b0a31051a8a37083bf P 9438789dc61f4501ad1d4ceaeccee458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:47.177208 3450 ts_tablet_manager.cc:1398] T 7a542c07a3f841b0a31051a8a37083bf P 9438789dc61f4501ad1d4ceaeccee458: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:47.177947 31471 tablet_copy_service.cc:342] P 6a6fcc099d1340c383e5a429643d5b7c: Request end of tablet copy session 9438789dc61f4501ad1d4ceaeccee458-7a542c07a3f841b0a31051a8a37083bf received from {username='slave'} at 127.23.191.70:52881
I20240501 14:09:47.178022 31471 tablet_copy_service.cc:433] P 6a6fcc099d1340c383e5a429643d5b7c: ending tablet copy session 9438789dc61f4501ad1d4ceaeccee458-7a542c07a3f841b0a31051a8a37083bf on tablet 7a542c07a3f841b0a31051a8a37083bf with peer 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:47.179402 3524 ts_tablet_manager.cc:1888] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:09:47.181787 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad0dac76fd0c4ae6b8e0549376c98e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.184343 3524 tablet_copy_client.cc:783] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:47.184825 3248 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:47.185060 3524 tablet_copy_client.cc:647] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:47.186456 3524 tablet_copy_client.cc:523] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:47.188149 3524 tablet_bootstrap.cc:492] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:47.191933 3248 tablet_bootstrap.cc:492] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:47.192168 3248 ts_tablet_manager.cc:1367] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.048s user 0.029s sys 0.000s
I20240501 14:09:47.192639 3248 raft_consensus.cc:348] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:47.192785 3248 raft_consensus.cc:729] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:47.192915 3248 consensus_queue.cc:260] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:47.193202 3248 ts_tablet_manager.cc:1398] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.193675 31471 tablet_copy_service.cc:342] P 6a6fcc099d1340c383e5a429643d5b7c: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-7a542c07a3f841b0a31051a8a37083bf received from {username='slave'} at 127.23.191.71:41729
I20240501 14:09:47.193769 31471 tablet_copy_service.cc:433] P 6a6fcc099d1340c383e5a429643d5b7c: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-7a542c07a3f841b0a31051a8a37083bf on tablet 7a542c07a3f841b0a31051a8a37083bf with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:47.217883 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.218075 31298 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:47.222187 31318 consensus_queue.cc:237] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:47.223500 472 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:47.224018 734 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:47.224360 3258 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:09:47.226375 31256 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d994081c3414f779ba2cae431cb1d97. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.227089 3258 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:47.227963 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:47.229359 31561 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:47.230343 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.233886 3211 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:47.234175 3211 tablet_bootstrap.cc:492] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Bootstrap complete.
I20240501 14:09:47.234364 3211 ts_tablet_manager.cc:1367] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Time spent bootstrapping tablet: real 0.111s user 0.027s sys 0.000s
I20240501 14:09:47.234838 3211 raft_consensus.cc:348] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } }
I20240501 14:09:47.234964 3211 raft_consensus.cc:729] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0caf2974041408bb32362bfc4284eaa, State: Initialized, Role: LEARNER
I20240501 14:09:47.235059 3211 consensus_queue.cc:260] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } }
I20240501 14:09:47.235347 3211 ts_tablet_manager.cc:1398] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:47.235782 31602 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session a0caf2974041408bb32362bfc4284eaa-72ad9c714b2b4fff94063d2838835173 received from {username='slave'} at 127.23.191.71:41541
I20240501 14:09:47.235862 31602 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session a0caf2974041408bb32362bfc4284eaa-72ad9c714b2b4fff94063d2838835173 on tablet 72ad9c714b2b4fff94063d2838835173 with peer a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:47.238394 3258 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:47.240512 734 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:47.241803 31042 catalog_manager.cc:5484] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 9 to 10, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:47.245110 473 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:47.273535 3524 tablet_bootstrap.cc:492] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:47.273908 3524 tablet_bootstrap.cc:492] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap complete.
I20240501 14:09:47.274112 3524 ts_tablet_manager.cc:1367] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.086s user 0.026s sys 0.000s
I20240501 14:09:47.274621 3524 raft_consensus.cc:348] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } }
I20240501 14:09:47.274760 3524 raft_consensus.cc:729] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: LEARNER
I20240501 14:09:47.274845 3524 consensus_queue.cc:260] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } }
I20240501 14:09:47.275727 3524 ts_tablet_manager.cc:1398] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:47.276192 31471 tablet_copy_service.cc:342] P 6a6fcc099d1340c383e5a429643d5b7c: Request end of tablet copy session e7eada5032fb45c3a5e3196e2b3faaab-ad0dac76fd0c4ae6b8e0549376c98e5d received from {username='slave'} at 127.23.191.68:50033
I20240501 14:09:47.276279 31471 tablet_copy_service.cc:433] P 6a6fcc099d1340c383e5a429643d5b7c: ending tablet copy session e7eada5032fb45c3a5e3196e2b3faaab-ad0dac76fd0c4ae6b8e0549376c98e5d on tablet ad0dac76fd0c4ae6b8e0549376c98e5d with peer e7eada5032fb45c3a5e3196e2b3faaab
I20240501 14:09:47.280542 31581 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54"
dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43370
I20240501 14:09:47.280699 31581 raft_consensus.cc:595] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:47.283150 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.286442 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.292735 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:47.310276 3602 ts_tablet_manager.cc:902] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a63e0dd1ba2e4a8db8ace9718a700c32: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
F20240501 14:09:47.310487 3602 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0399a419a25c4981a2ee90d76cbf59b1.queue_time_us, tablet-download-pool-0399a419a25c4981a2ee90d76cbf59b1.queue_time_us) (tablet-download-pool-0399a419a25c4981a2ee90d76cbf59b1.run_cpu_time_us, tablet-download-pool-0399a419a25c4981a2ee90d76cbf59b1.run_cpu_time_us) (tablet-download-pool-0399a419a25c4981a2ee90d76cbf59b1.run_wall_time_us, tablet-download-pool-0399a419a25c4981a2ee90d76cbf59b1.run_wall_time_us) (tablet-download-pool-2e68be1521964ba8b864801fdf3b0df3.queue_time_us, tablet-download-pool-2e68be1521964ba8b864801fdf3b0df3.queue_time_us) (tablet-download-pool-2e68be1521964ba8b864801fdf3b0df3.run_cpu_time_us, tablet-download-pool-2e68be1521964ba8b864801fdf3b0df3.run_cpu_time_us) (tablet-download-pool-2e68be1521964ba8b864801fdf3b0df3.run_wall_time_us, tablet-download-pool-2e68be1521964ba8b864801fdf3b0df3.run_wall_time_us) (tablet-download-pool-4603d1b1b1d441b1b0be0464e2f2c9e2.queue_time_us, tablet-download-pool-4603d1b1b1d441b1b0be0464e2f2c9e2.queue_time_us) (tablet-download-pool-5aa7bdac7bd1408cbf41d894d40aebd4.queue_time_us, tablet-download-pool-5aa7bdac7bd1408cbf41d894d40aebd4.queue_time_us) (tablet-download-pool-5aa7bdac7bd1408cbf41d894d40aebd4.run_cpu_time_us, tablet-download-pool-5aa7bdac7bd1408cbf41d894d40aebd4.run_cpu_time_us) (tablet-download-pool-5aa7bdac7bd1408cbf41d894d40aebd4.run_wall_time_us, tablet-download-pool-5aa7bdac7bd1408cbf41d894d40aebd4.run_wall_time_us) (tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.queue_time_us, tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.queue_time_us) (tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.run_cpu_time_us, tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.run_cpu_time_us) (tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.run_wall_time_us, tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.run_wall_time_us) (tablet-download-pool-6f8bd03c05d84fdaaf138a6d0608e30e.queue_time_us, tablet-download-pool-6f8bd03c05d84fdaaf138a6d0608e30e.queue_time_us) (tablet-download-pool-6f8bd03c05d84fdaaf138a6d0608e30e.run_cpu_time_us, tablet-download-pool-6f8bd03c05d84fdaaf138a6d0608e30e.run_cpu_time_us) (tablet-download-pool-6f8bd03c05d84fdaaf138a6d0608e30e.run_wall_time_us, tablet-download-pool-6f8bd03c05d84fdaaf138a6d0608e30e.run_wall_time_us) (tablet-download-pool-72ad9c714b2b4fff94063d2838835173.queue_time_us, tablet-download-pool-72ad9c714b2b4fff94063d2838835173.queue_time_us) (tablet-download-pool-72ad9c714b2b4fff94063d2838835173.run_cpu_time_us, tablet-download-pool-72ad9c714b2b4fff94063d2838835173.run_cpu_time_us) (tablet-download-pool-72ad9c714b2b4fff94063d2838835173.run_wall_time_us, tablet-download-pool-72ad9c714b2b4fff94063d2838835173.run_wall_time_us) (tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.queue_time_us, tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.queue_time_us) (tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.run_cpu_time_us, tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.run_cpu_time_us) (tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.run_wall_time_us, tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.run_wall_time_us) (tablet-download-pool-8f91909d3b1b46e9b858ce5c5b2e7975.queue_time_us, tablet-download-pool-8f91909d3b1b46e9b858ce5c5b2e7975.queue_time_us) (tablet-download-pool-8f91909d3b1b46e9b858ce5c5b2e7975.run_cpu_time_us, tablet-download-pool-8f91909d3b1b46e9b858ce5c5b2e7975.run_cpu_time_us) (tablet-download-pool-8f91909d3b1b46e9b858ce5c5b2e7975.run_wall_time_us, tablet-download-pool-8f91909d3b1b46e9b858ce5c5b2e7975.run_wall_time_us) (tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.queue_time_us, tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.queue_time_us) (tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.run_cpu_time_us, tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.run_cpu_time_us) (tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.run_wall_time_us, tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.run_wall_time_us) (tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.queue_time_us, tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.queue_time_us) (tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_cpu_time_us, tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_cpu_time_us) (tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_wall_time_us, tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_wall_time_us) (tablet-download-pool-bca09920a7904362bc546916e3f6f5ac.queue_time_us, tablet-download-pool-bca09920a7904362bc546916e3f6f5ac.queue_time_us) (tablet-download-pool-bca09920a7904362bc546916e3f6f5ac.run_cpu_time_us, tablet-download-pool-bca09920a7904362bc546916e3f6f5ac.run_cpu_time_us) (tablet-download-pool-bca09920a7904362bc546916e3f6f5ac.run_wall_time_us, tablet-download-pool-bca09920a7904362bc546916e3f6f5ac.run_wall_time_us) (tablet-download-pool-c082ccc777ba4f9a937931f707645e54.queue_time_us, tablet-download-pool-c082ccc777ba4f9a937931f707645e54.queue_time_us) (tablet-download-pool-c082ccc777ba4f9a937931f707645e54.run_cpu_time_us, tablet-download-pool-c082ccc777ba4f9a937931f707645e54.run_cpu_time_us) (tablet-download-pool-c082ccc777ba4f9a937931f707645e54.run_wall_time_us, tablet-download-pool-c082ccc777ba4f9a937931f707645e54.run_wall_time_us) (tablet-download-pool-c0b7ede9ffaf496e9b106b5d05ed25a5.queue_time_us, tablet-download-pool-c0b7ede9ffaf496e9b106b5d05ed25a5.queue_time_us) (tablet-download-pool-c0b7ede9ffaf496e9b106b5d05ed25a5.run_cpu_time_us, tablet-download-pool-c0b7ede9ffaf496e9b106b5d05ed25a5.run_cpu_time_us) (tablet-download-pool-c0b7ede9ffaf496e9b106b5d05ed25a5.run_wall_time_us, tablet-download-pool-c0b7ede9ffaf496e9b106b5d05ed25a5.run_wall_time_us) (tablet-download-pool-d265eff200b645bfac8f8906654385f6.queue_time_us, tablet-download-pool-d265eff200b645bfac8f8906654385f6.queue_time_us) (tablet-download-pool-d265eff200b645bfac8f8906654385f6.run_cpu_time_us, tablet-download-pool-d265eff200b645bfac8f8906654385f6.run_cpu_time_us) (tablet-download-pool-d265eff200b645bfac8f8906654385f6.run_wall_time_us, tablet-download-pool-d265eff200b645bfac8f8906654385f6.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fc4d563adcd google::LogMessage::Fail() at ??:0
I20240501 14:09:47.316555 31581 consensus_queue.cc:237] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } }
@ 0x7fc4d563eb93 google::LogMessage::SendToLog() at ??:0
I20240501 14:09:47.318395 31188 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:47.318830 2877 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
@ 0x7fc4d563a7cc google::LogMessage::Flush() at ??:0
I20240501 14:09:47.319824 865 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:47.320973 2877 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
@ 0x7fc4d563bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
W20240501 14:09:47.321472 31516 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): Couldn't send request to peer 6a6fcc099d1340c383e5a429643d5b7c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.323556 2891 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } } }
@ 0x7fc4d6aa4044 kudu::TraceMetrics::InternName() at ??:0
I20240501 14:09:47.325359 865 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } } }
@ 0x7fc4d6a88f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240501 14:09:47.328246 31047 catalog_manager.cc:5484] T d265eff200b645bfac8f8906654385f6 P a63e0dd1ba2e4a8db8ace9718a700c32 reported cstate change: config changed from index 6 to 7, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } } }
I20240501 14:09:47.330397 31188 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } } }
@ 0x7fc4d6a86efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fc4d963977a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fc4d9642a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fc4d96c376f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fc4d96d74f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fc4d96c2226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fc4d96d1c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56352cf3f75c std::function<>::operator()() at ??:0
@ 0x7fc4d6a8be06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc4d6a8c6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc4d6a8e03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56352cf3f75c std::function<>::operator()() at ??:0
I20240501 14:09:47.366868 3524 ts_tablet_manager.cc:902] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:47.367317 3524 tablet_copy_client.cc:308] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:47.367554 31188 consensus_queue.cc:237] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:47.367738 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet 8f91909d3b1b46e9b858ce5c5b2e7975 from peer e7eada5032fb45c3a5e3196e2b3faaab ({username='slave'} at 127.23.191.68:40875)
I20240501 14:09:47.367839 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet 8f91909d3b1b46e9b858ce5c5b2e7975 from peer e7eada5032fb45c3a5e3196e2b3faaab at {username='slave'} at 127.23.191.68:40875: session id = e7eada5032fb45c3a5e3196e2b3faaab-8f91909d3b1b46e9b858ce5c5b2e7975
I20240501 14:09:47.368789 31340 tablet_copy_source_session.cc:205] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 14:09:47.369184 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.369196 31318 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:47.369561 2405 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:47.370303 3524 ts_tablet_manager.cc:1875] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:09:47.371132 865 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:47.371534 2405 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:47.372951 3524 ts_tablet_manager.cc:1888] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:09:47.373098 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f91909d3b1b46e9b858ce5c5b2e7975. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.373733 31298 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:47.383122 586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.385075 31429 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:47.396752 841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.397112 3524 tablet_copy_client.cc:783] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:47.397305 3524 tablet_copy_client.cc:647] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:47.399024 2405 raft_consensus.cc:1050] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 to VOTER
I20240501 14:09:47.399418 2405 consensus_queue.cc:237] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:47.400785 472 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:47.401197 3439 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:47.402669 868 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P a63e0dd1ba2e4a8db8ace9718a700c32 [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:47.403383 603 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:47.403353 3440 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:47.403903 3440 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:47.404460 3524 tablet_copy_client.cc:523] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:47.405980 3524 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
@ 0x7fc4d6a7b807 kudu::Thread::SuperviseThread() at ??:0
I20240501 14:09:47.415748 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7fc4d72b96db start_thread at ??:0
I20240501 14:09:47.417618 2877 raft_consensus.cc:979] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab: : Instructing follower 9438789dc61f4501ad1d4ceaeccee458 to start an election
I20240501 14:09:47.417706 2877 raft_consensus.cc:1067] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Signalling peer 9438789dc61f4501ad1d4ceaeccee458 to start an election
I20240501 14:09:47.418308 602 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54"
dest_uuid: "9438789dc61f4501ad1d4ceaeccee458"
from {username='slave'} at 127.23.191.68:38295
I20240501 14:09:47.418469 602 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:47.418563 602 raft_consensus.cc:3037] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Advancing to term 2
@ 0x7fc4d47f171f clone at ??:0
I20240501 14:09:47.419610 602 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:47.420763 602 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [CANDIDATE]: Term 2 election: Requested vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429)
I20240501 14:09:47.420933 31317 consensus_queue.cc:237] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } }
I20240501 14:09:47.421844 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "9438789dc61f4501ad1d4ceaeccee458" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:47.421955 31581 raft_consensus.cc:3032] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:47.421994 31581 raft_consensus.cc:729] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Running, Role: LEADER
I20240501 14:09:47.422139 31581 consensus_queue.cc:260] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:47.422323 31581 raft_consensus.cc:3037] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:47.422536 472 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:47.423183 2513 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:47.423305 31581 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9438789dc61f4501ad1d4ceaeccee458 in term 2.
W20240501 14:09:47.423679 31253 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): Couldn't send request to peer 0e4622d80cc94f12b492bdd7597a61e1. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.424371 601 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:47.424729 2513 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:47.425102 2513 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
I20240501 14:09:47.425660 472 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
I20240501 14:09:47.426424 31561 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:47.427851 472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "9438789dc61f4501ad1d4ceaeccee458" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "b934fd09590a40ada094b9715d990a00"
I20240501 14:09:47.427997 472 raft_consensus.cc:3037] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:47.428557 31040 catalog_manager.cc:5484] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 11 to 12, NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
I20240501 14:09:47.429248 3440 raft_consensus.cc:1050] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER a0caf2974041408bb32362bfc4284eaa to VOTER
I20240501 14:09:47.429451 472 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9438789dc61f4501ad1d4ceaeccee458 in term 2.
I20240501 14:09:47.429483 601 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } } }
I20240501 14:09:47.429694 3440 consensus_queue.cc:237] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:47.429754 541 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:47.429968 3474 raft_consensus.cc:2781] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:47.430048 3474 raft_consensus.cc:686] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [term 2 LEADER]: Becoming Leader. State: Replica: 9438789dc61f4501ad1d4ceaeccee458, State: Running, Role: LEADER
I20240501 14:09:47.430300 3474 consensus_queue.cc:237] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:47.431288 734 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:09:47.431481 473 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:09:47.431766 3439 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:47.432080 3439 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:47.432898 31039 catalog_manager.cc:5484] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 reported cstate change: term changed from 1 to 2, leader changed from e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) to 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70). New cstate: current_term: 2 leader_uuid: "9438789dc61f4501ad1d4ceaeccee458" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:47.444651 3524 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:47.445449 3524 tablet_bootstrap.cc:492] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap complete.
I20240501 14:09:47.445644 3524 ts_tablet_manager.cc:1367] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.040s user 0.023s sys 0.005s
I20240501 14:09:47.446174 3524 raft_consensus.cc:348] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } }
I20240501 14:09:47.446408 3524 raft_consensus.cc:729] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: LEARNER
I20240501 14:09:47.446516 3524 consensus_queue.cc:260] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } }
I20240501 14:09:47.446779 3524 ts_tablet_manager.cc:1398] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:47.448019 3420 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:47.448535 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session e7eada5032fb45c3a5e3196e2b3faaab-8f91909d3b1b46e9b858ce5c5b2e7975 received from {username='slave'} at 127.23.191.68:40875
I20240501 14:09:47.448598 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session e7eada5032fb45c3a5e3196e2b3faaab-8f91909d3b1b46e9b858ce5c5b2e7975 on tablet 8f91909d3b1b46e9b858ce5c5b2e7975 with peer e7eada5032fb45c3a5e3196e2b3faaab
I20240501 14:09:47.448475 734 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:47.450918 3419 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:47.452095 3419 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:47.454383 472 raft_consensus.cc:2932] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:47.457087 603 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:47.460222 472 raft_consensus.cc:2932] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:47.460448 31318 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:47.461666 31039 catalog_manager.cc:5484] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 13 to 14, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:47.462100 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.463724 31041 catalog_manager.cc:5484] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 9 to 10, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:47.468490 31040 catalog_manager.cc:5484] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 10 to 11, a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } }
I20240501 14:09:47.472163 31318 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea66fe3d0da04d18b961927a9c988a20"
dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46278
I20240501 14:09:47.472326 31318 raft_consensus.cc:595] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:47.477532 31318 raft_consensus.cc:1203] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240501 14:09:47.484771 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:47.506671 2877 raft_consensus.cc:1050] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER a0caf2974041408bb32362bfc4284eaa to VOTER
I20240501 14:09:47.507052 2877 consensus_queue.cc:237] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:47.508445 734 raft_consensus.cc:1261] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:47.508543 31448 raft_consensus.cc:1261] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:47.508790 31188 raft_consensus.cc:1261] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:47.508966 1589 consensus_queue.cc:1035] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:47.509155 2683 consensus_queue.cc:1035] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:47.509295 2891 consensus_queue.cc:1035] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:47.514624 2683 raft_consensus.cc:2932] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:47.517270 31448 raft_consensus.cc:2932] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:47.517596 31040 catalog_manager.cc:5484] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:47.519397 734 raft_consensus.cc:2932] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:47.523151 31188 raft_consensus.cc:2932] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:47.530413 603 raft_consensus.cc:1203] T f26d16499ab54a3e9059f6e4ff041177 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:09:47.535804 3721 ts_tablet_manager.cc:902] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:47.536969 3721 tablet_copy_client.cc:308] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:47.537503 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet 6f8bd03c05d84fdaaf138a6d0608e30e from peer 6a6fcc099d1340c383e5a429643d5b7c ({username='slave'} at 127.23.191.67:35893)
I20240501 14:09:47.537609 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet 6f8bd03c05d84fdaaf138a6d0608e30e from peer 6a6fcc099d1340c383e5a429643d5b7c at {username='slave'} at 127.23.191.67:35893: session id = 6a6fcc099d1340c383e5a429643d5b7c-6f8bd03c05d84fdaaf138a6d0608e30e
I20240501 14:09:47.538267 31209 tablet_copy_source_session.cc:205] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:47.539139 3721 ts_tablet_manager.cc:1875] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:09:47.541112 3721 ts_tablet_manager.cc:1888] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:09:47.541296 3721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8bd03c05d84fdaaf138a6d0608e30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.543128 3721 tablet_copy_client.cc:783] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:47.543298 3721 tablet_copy_client.cc:647] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:47.544632 3721 tablet_copy_client.cc:523] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:47.546298 3721 tablet_bootstrap.cc:492] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:47.552213 2877 raft_consensus.cc:1050] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 to VOTER
I20240501 14:09:47.552677 2877 consensus_queue.cc:237] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } }
I20240501 14:09:47.554394 472 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:47.555115 2683 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:47.555527 603 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:47.555904 2683 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:47.556591 31188 raft_consensus.cc:1203] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:09:47.556605 31189 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:47.557556 2891 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:47.559785 2683 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:47.588701 3721 tablet_bootstrap.cc:492] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:47.594863 3721 tablet_bootstrap.cc:492] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap complete.
I20240501 14:09:47.595117 3721 ts_tablet_manager.cc:1367] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.049s user 0.027s sys 0.000s
I20240501 14:09:47.561261 603 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:47.625522 3721 raft_consensus.cc:348] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } }
I20240501 14:09:47.625669 3721 raft_consensus.cc:729] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: LEARNER
I20240501 14:09:47.625764 3721 consensus_queue.cc:260] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } }
I20240501 14:09:47.626042 3721 ts_tablet_manager.cc:1398] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.615574 602 raft_consensus.cc:1203] T 7a542c07a3f841b0a31051a8a37083bf P 9438789dc61f4501ad1d4ceaeccee458 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:09:47.628988 31581 raft_consensus.cc:1203] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:09:47.630867 734 raft_consensus.cc:1203] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:09:47.560456 31189 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:47.633077 2891 raft_consensus.cc:1050] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER a0caf2974041408bb32362bfc4284eaa to VOTER
I20240501 14:09:47.627290 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session 6a6fcc099d1340c383e5a429643d5b7c-6f8bd03c05d84fdaaf138a6d0608e30e received from {username='slave'} at 127.23.191.67:35893
I20240501 14:09:47.634470 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session 6a6fcc099d1340c383e5a429643d5b7c-6f8bd03c05d84fdaaf138a6d0608e30e on tablet 6f8bd03c05d84fdaaf138a6d0608e30e with peer 6a6fcc099d1340c383e5a429643d5b7c
I20240501 14:09:47.634792 2891 consensus_queue.cc:237] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:47.636509 31448 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:47.636986 2453 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:47.637770 734 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:47.639091 2453 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:47.647859 734 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:47.647241 2453 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:47.562904 472 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:47.593629 3210 raft_consensus.cc:979] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a: : Instructing follower 6a6fcc099d1340c383e5a429643d5b7c to start an election
I20240501 14:09:47.609897 31318 consensus_queue.cc:237] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:09:47.660418 3210 raft_consensus.cc:1067] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Signalling peer 6a6fcc099d1340c383e5a429643d5b7c to start an election
I20240501 14:09:47.563618 31041 catalog_manager.cc:5484] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 10 to 11, 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:47.661334 31448 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea66fe3d0da04d18b961927a9c988a20"
dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
from {username='slave'} at 127.23.191.66:53563
I20240501 14:09:47.661499 31448 raft_consensus.cc:482] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:47.661568 31448 raft_consensus.cc:3037] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:47.662487 31450 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:47.662580 31448 raft_consensus.cc:504] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } }
I20240501 14:09:47.662941 31448 leader_election.cc:288] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 2 election: Requested vote from peers 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
I20240501 14:09:47.664145 737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea66fe3d0da04d18b961927a9c988a20" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "a0caf2974041408bb32362bfc4284eaa"
I20240501 14:09:47.664326 737 raft_consensus.cc:3037] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:47.665464 737 raft_consensus.cc:2454] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 2.
I20240501 14:09:47.666427 31385 leader_election.cc:302] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a6fcc099d1340c383e5a429643d5b7c, a0caf2974041408bb32362bfc4284eaa; no voters:
I20240501 14:09:47.666591 2419 raft_consensus.cc:2781] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:47.666666 2419 raft_consensus.cc:686] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 2 LEADER]: Becoming Leader. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Running, Role: LEADER
I20240501 14:09:47.666793 2419 consensus_queue.cc:237] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } }
I20240501 14:09:47.667685 2513 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:47.668466 31317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea66fe3d0da04d18b961927a9c988a20" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "30e0b8849e3841ca9ab0de01e719ee4a"
I20240501 14:09:47.668570 31317 raft_consensus.cc:3032] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:47.668607 31317 raft_consensus.cc:729] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Running, Role: LEADER
I20240501 14:09:47.668758 31317 consensus_queue.cc:260] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } }
I20240501 14:09:47.668980 31317 raft_consensus.cc:3037] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:47.669394 31039 catalog_manager.cc:5484] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: term changed from 1 to 2, leader changed from 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) to 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67). New cstate: current_term: 2 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:47.670001 31317 raft_consensus.cc:2454] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 2.
I20240501 14:09:47.670770 31581 consensus_queue.cc:237] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } }
I20240501 14:09:47.672253 31189 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:47.673007 31046 catalog_manager.cc:5484] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 7 to 8, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:47.673178 2877 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:47.673913 473 raft_consensus.cc:1261] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:47.675026 734 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:47.675549 2513 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:47.676344 2683 consensus_queue.cc:1035] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:47.676792 31450 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
W20240501 14:09:47.676970 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4603d1b1b1d441b1b0be0464e2f2c9e2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.676760 2513 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:47.678622 2683 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:47.679620 31450 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:47.681875 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d2c173b2c32494eb16e46a51ca36e31 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:47.682724 31042 catalog_manager.cc:5484] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 7 to 8, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:47.684429 734 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:09:47.684788 472 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:47.687342 31047 catalog_manager.cc:5484] T 9d2c173b2c32494eb16e46a51ca36e31 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 11 to 12, VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:47.688122 31189 raft_consensus.cc:2932] T 9d2c173b2c32494eb16e46a51ca36e31 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:47.695133 586 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d2c173b2c32494eb16e46a51ca36e31 with delete_type TABLET_DATA_TOMBSTONED (TS 9438789dc61f4501ad1d4ceaeccee458 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:46552
I20240501 14:09:47.697839 734 raft_consensus.cc:1203] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240501 14:09:47.717155 3841 tablet_replica.cc:330] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458: stopping tablet replica
I20240501 14:09:47.717335 3841 raft_consensus.cc:2229] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:47.717528 3841 raft_consensus.cc:2258] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:47.719112 3841 ts_tablet_manager.cc:1875] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:09:47.719897 31387 connection.cc:535] client connection to 127.23.191.72:39799 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:47.720052 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:47.719969 31518 connection.cc:535] client connection to 127.23.191.72:39799 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:47.720512 3473 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:09:47.427864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.191.72:39799 (local address 127.23.191.70:58073)
0501 14:09:47.427958 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0501 14:09:47.427971 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 14:09:47.428038 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:09:47.719951 (+291913us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":25}
W20240501 14:09:47.720577 31125 connection.cc:535] client connection to 127.23.191.72:39799 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:47.720041 31256 connection.cc:535] client connection to 127.23.191.72:39799 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:47.720691 542 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:47.720697 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:47.720937 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:47.721331 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:47.724421 542 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:47.725802 3841 ts_tablet_manager.cc:1888] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 14:09:47.725947 3841 log.cc:1196] T 9d2c173b2c32494eb16e46a51ca36e31 P 9438789dc61f4501ad1d4ceaeccee458: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-5/wal/wals/9d2c173b2c32494eb16e46a51ca36e31
I20240501 14:09:47.726868 31034 catalog_manager.cc:4830] TS 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): tablet 9d2c173b2c32494eb16e46a51ca36e31 (table rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]) successfully deleted
W20240501 14:09:47.728039 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:47.728109 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:47.731820 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:47.731941 31256 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:47.731998 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:47.737187 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:47.737260 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:47.739903 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:47.739965 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:47.739995 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:47.740033 31125 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:47.740065 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:47.740100 31125 consensus_peers.cc:485] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.772330 3721 ts_tablet_manager.cc:902] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
I20240501 14:09:47.772904 3721 tablet_copy_client.cc:308] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.68:45349
I20240501 14:09:47.773463 31602 tablet_copy_service.cc:140] P e7eada5032fb45c3a5e3196e2b3faaab: Received BeginTabletCopySession request for tablet d265eff200b645bfac8f8906654385f6 from peer 6a6fcc099d1340c383e5a429643d5b7c ({username='slave'} at 127.23.191.67:51267)
I20240501 14:09:47.773555 31602 tablet_copy_service.cc:161] P e7eada5032fb45c3a5e3196e2b3faaab: Beginning new tablet copy session on tablet d265eff200b645bfac8f8906654385f6 from peer 6a6fcc099d1340c383e5a429643d5b7c at {username='slave'} at 127.23.191.67:51267: session id = 6a6fcc099d1340c383e5a429643d5b7c-d265eff200b645bfac8f8906654385f6
I20240501 14:09:47.774513 31602 tablet_copy_source_session.cc:205] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:47.775229 3721 ts_tablet_manager.cc:1875] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:09:47.779170 3721 ts_tablet_manager.cc:1888] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:09:47.779335 3721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d265eff200b645bfac8f8906654385f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.780900 3721 tablet_copy_client.cc:783] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:47.781034 3721 tablet_copy_client.cc:647] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:47.782557 3721 tablet_copy_client.cc:523] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:47.783735 3721 tablet_bootstrap.cc:492] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:09:47.792279 31581 raft_consensus.cc:1203] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240501 14:09:47.810223 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.812934 31429 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:47.813175 586 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:47.820034 453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.820348 31561 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:47.829907 542 consensus_peers.cc:485] T c082ccc777ba4f9a937931f707645e54 P 9438789dc61f4501ad1d4ceaeccee458 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.835877 3721 tablet_bootstrap.cc:492] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:47.836171 3721 tablet_bootstrap.cc:492] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap complete.
I20240501 14:09:47.836359 3721 ts_tablet_manager.cc:1367] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.053s user 0.025s sys 0.000s
I20240501 14:09:47.839213 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.836791 3721 raft_consensus.cc:348] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } }
I20240501 14:09:47.841010 3721 raft_consensus.cc:729] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: LEARNER
I20240501 14:09:47.841128 3721 consensus_queue.cc:260] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } }
I20240501 14:09:47.842520 3721 ts_tablet_manager.cc:1398] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:09:47.843777 3991 ts_tablet_manager.cc:902] T 5d994081c3414f779ba2cae431cb1d97 P 9438789dc61f4501ad1d4ceaeccee458: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
F20240501 14:09:47.843932 3991 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0399a419a25c4981a2ee90d76cbf59b1.queue_time_us, tablet-download-pool-0399a419a25c4981a2ee90d76cbf59b1.queue_time_us) (tablet-download-pool-0399a419a25c4981a2ee90d76cbf59b1.run_cpu_time_us, tablet-download-pool-0399a419a25c4981a2ee90d76cbf59b1.run_cpu_time_us) (tablet-download-pool-0399a419a25c4981a2ee90d76cbf59b1.run_wall_time_us, tablet-download-pool-0399a419a25c4981a2ee90d76cbf59b1.run_wall_time_us) (tablet-download-pool-28013d7df97943e09c27c3bb1794ee80.queue_time_us, tablet-download-pool-28013d7df97943e09c27c3bb1794ee80.queue_time_us) (tablet-download-pool-28013d7df97943e09c27c3bb1794ee80.run_cpu_time_us, tablet-download-pool-28013d7df97943e09c27c3bb1794ee80.run_cpu_time_us) (tablet-download-pool-28013d7df97943e09c27c3bb1794ee80.run_wall_time_us, tablet-download-pool-28013d7df97943e09c27c3bb1794ee80.run_wall_time_us) (tablet-download-pool-3c8200d9c20e4140906c71c746badb94.queue_time_us, tablet-download-pool-3c8200d9c20e4140906c71c746badb94.queue_time_us) (tablet-download-pool-3c8200d9c20e4140906c71c746badb94.run_cpu_time_us, tablet-download-pool-3c8200d9c20e4140906c71c746badb94.run_cpu_time_us) (tablet-download-pool-3c8200d9c20e4140906c71c746badb94.run_wall_time_us, tablet-download-pool-3c8200d9c20e4140906c71c746badb94.run_wall_time_us) (tablet-download-pool-5d994081c3414f779ba2cae431cb1d97.queue_time_us, tablet-download-pool-5d994081c3414f779ba2cae431cb1d97.queue_time_us) (tablet-download-pool-6bcd6a2f3c014d5ca0e32e16cdbc4edc.queue_time_us, tablet-download-pool-6bcd6a2f3c014d5ca0e32e16cdbc4edc.queue_time_us) (tablet-download-pool-6bcd6a2f3c014d5ca0e32e16cdbc4edc.run_cpu_time_us, tablet-download-pool-6bcd6a2f3c014d5ca0e32e16cdbc4edc.run_cpu_time_us) (tablet-download-pool-6bcd6a2f3c014d5ca0e32e16cdbc4edc.run_wall_time_us, tablet-download-pool-6bcd6a2f3c014d5ca0e32e16cdbc4edc.run_wall_time_us) (tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.queue_time_us, tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.queue_time_us) (tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.run_cpu_time_us, tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.run_cpu_time_us) (tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.run_wall_time_us, tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.run_wall_time_us) (tablet-download-pool-8f91909d3b1b46e9b858ce5c5b2e7975.queue_time_us, tablet-download-pool-8f91909d3b1b46e9b858ce5c5b2e7975.queue_time_us) (tablet-download-pool-8f91909d3b1b46e9b858ce5c5b2e7975.run_cpu_time_us, tablet-download-pool-8f91909d3b1b46e9b858ce5c5b2e7975.run_cpu_time_us) (tablet-download-pool-8f91909d3b1b46e9b858ce5c5b2e7975.run_wall_time_us, tablet-download-pool-8f91909d3b1b46e9b858ce5c5b2e7975.run_wall_time_us) (tablet-download-pool-96f5e4e25a674a99a6762757f0ab1545.queue_time_us, tablet-download-pool-96f5e4e25a674a99a6762757f0ab1545.queue_time_us) (tablet-download-pool-96f5e4e25a674a99a6762757f0ab1545.run_cpu_time_us, tablet-download-pool-96f5e4e25a674a99a6762757f0ab1545.run_cpu_time_us) (tablet-download-pool-96f5e4e25a674a99a6762757f0ab1545.run_wall_time_us, tablet-download-pool-96f5e4e25a674a99a6762757f0ab1545.run_wall_time_us) (tablet-download-pool-9d2c173b2c32494eb16e46a51ca36e31.queue_time_us, tablet-download-pool-9d2c173b2c32494eb16e46a51ca36e31.queue_time_us) (tablet-download-pool-9d2c173b2c32494eb16e46a51ca36e31.run_cpu_time_us, tablet-download-pool-9d2c173b2c32494eb16e46a51ca36e31.run_cpu_time_us) (tablet-download-pool-9d2c173b2c32494eb16e46a51ca36e31.run_wall_time_us, tablet-download-pool-9d2c173b2c32494eb16e46a51ca36e31.run_wall_time_us) (tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.queue_time_us, tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.queue_time_us) (tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.run_cpu_time_us, tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.run_cpu_time_us) (tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.run_wall_time_us, tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.run_wall_time_us) (tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.queue_time_us, tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.queue_time_us) (tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_cpu_time_us, tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_cpu_time_us) (tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_wall_time_us, tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_wall_time_us) (tablet-download-pool-c082ccc777ba4f9a937931f707645e54.queue_time_us, tablet-download-pool-c082ccc777ba4f9a937931f707645e54.queue_time_us) (tablet-download-pool-c082ccc777ba4f9a937931f707645e54.run_cpu_time_us, tablet-download-pool-c082ccc777ba4f9a937931f707645e54.run_cpu_time_us) (tablet-download-pool-c082ccc777ba4f9a937931f707645e54.run_wall_time_us, tablet-download-pool-c082ccc777ba4f9a937931f707645e54.run_wall_time_us) (tablet-download-pool-c0b7ede9ffaf496e9b106b5d05ed25a5.queue_time_us, tablet-download-pool-c0b7ede9ffaf496e9b106b5d05ed25a5.queue_time_us) (tablet-download-pool-c0b7ede9ffaf496e9b106b5d05ed25a5.run_cpu_time_us, tablet-download-pool-c0b7ede9ffaf496e9b106b5d05ed25a5.run_cpu_time_us) (tablet-download-pool-c0b7ede9ffaf496e9b106b5d05ed25a5.run_wall_time_us, tablet-download-pool-c0b7ede9ffaf496e9b106b5d05ed25a5.run_wall_time_us) (tablet-download-pool-f26d16499ab54a3e9059f6e4ff041177.queue_time_us, tablet-download-pool-f26d16499ab54a3e9059f6e4ff041177.queue_time_us) (tablet-download-pool-f26d16499ab54a3e9059f6e4ff041177.run_cpu_time_us, tablet-download-pool-f26d16499ab54a3e9059f6e4ff041177.run_cpu_time_us) (tablet-download-pool-f26d16499ab54a3e9059f6e4ff041177.run_wall_time_us, tablet-download-pool-f26d16499ab54a3e9059f6e4ff041177.run_wall_time_us) (tablet-download-pool-f8cc4d3985224178aa008b4c9de5625b.queue_time_us, tablet-download-pool-f8cc4d3985224178aa008b4c9de5625b.queue_time_us) (tablet-download-pool-f8cc4d3985224178aa008b4c9de5625b.run_cpu_time_us, tablet-download-pool-f8cc4d3985224178aa008b4c9de5625b.run_cpu_time_us) (tablet-download-pool-f8cc4d3985224178aa008b4c9de5625b.run_wall_time_us, tablet-download-pool-f8cc4d3985224178aa008b4c9de5625b.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 14:09:47.845032 31169 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:47.846009 31602 tablet_copy_service.cc:342] P e7eada5032fb45c3a5e3196e2b3faaab: Request end of tablet copy session 6a6fcc099d1340c383e5a429643d5b7c-d265eff200b645bfac8f8906654385f6 received from {username='slave'} at 127.23.191.67:51267
I20240501 14:09:47.846087 31602 tablet_copy_service.cc:433] P e7eada5032fb45c3a5e3196e2b3faaab: ending tablet copy session 6a6fcc099d1340c383e5a429643d5b7c-d265eff200b645bfac8f8906654385f6 on tablet d265eff200b645bfac8f8906654385f6 with peer 6a6fcc099d1340c383e5a429643d5b7c
@ 0x7f5a0c205dcd google::LogMessage::Fail() at ??:0
@ 0x7f5a0c209b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f5a0c2057cc google::LogMessage::Flush() at ??:0
@ 0x7f5a0c206f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5a0d66f044 kudu::TraceMetrics::InternName() at ??:0
I20240501 14:09:47.861073 31298 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
@ 0x7f5a0d653f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f5a0d651efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f5a1020477a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 14:09:47.873003 3992 ts_tablet_manager.cc:902] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:47.873463 3992 tablet_copy_client.cc:308] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:47.873839 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet 3c8200d9c20e4140906c71c746badb94 from peer 0e4622d80cc94f12b492bdd7597a61e1 ({username='slave'} at 127.23.191.65:46557)
I20240501 14:09:47.873925 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet 3c8200d9c20e4140906c71c746badb94 from peer 0e4622d80cc94f12b492bdd7597a61e1 at {username='slave'} at 127.23.191.65:46557: session id = 0e4622d80cc94f12b492bdd7597a61e1-3c8200d9c20e4140906c71c746badb94
I20240501 14:09:47.874768 31340 tablet_copy_source_session.cc:205] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:47.875413 3992 ts_tablet_manager.cc:1875] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: Deleting tablet data with delete state TABLET_DATA_COPYING
@ 0x7f5a1020da45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 14:09:47.888934 3992 ts_tablet_manager.cc:1888] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.10
I20240501 14:09:47.889109 3992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8200d9c20e4140906c71c746badb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.890369 3992 tablet_copy_client.cc:783] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:47.890491 3992 tablet_copy_client.cc:647] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: tablet copy: Starting download of 1 WAL segments...
W20240501 14:09:47.891762 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:47.892424 3992 tablet_copy_client.cc:523] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:47.893690 3992 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap starting.
@ 0x7f5a1028e76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f5a102a24f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f5a1028d226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f5a1029cc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x562062ec275c std::function<>::operator()() at ??:0
I20240501 14:09:47.909689 31189 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545"
dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50104
I20240501 14:09:47.909832 31189 raft_consensus.cc:595] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Received request to transfer leadership
@ 0x7f5a0d656e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5a0d6576ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f5a0d65903b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x562062ec275c std::function<>::operator()() at ??:0
I20240501 14:09:47.929294 917 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:09:41.357106 923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:41.357372 923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:41.361667 923 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:09:42.291184 923 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:09:42.800056 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 855c959aa35441b989d386caf8f8eded: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:42.800190 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01dc7a764a349c9a4a9180993e4fbc3: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:42.800231 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77c4973914b2412cbc293014310ba9c6: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:42.800282 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:42.800341 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd036c60a12148d2ada73beb29aedcbf: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:42.800451 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bd1074f581c447ee86e97b1ef3b8687f: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:42.800506 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:42.800550 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1f2efe25014e49d8b36ca467c0f40ef3: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:42.800614 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 351ff5b1095247928fbb77893616e4a9: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:42.800680 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbe6d56762d84933a7b7ddaf8c314b4f: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:42.800730 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 026cd477adec4547bee4fb8004a63348: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:42.800807 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77c4973914b2412cbc293014310ba9c6: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:42.811064 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bd1074f581c447ee86e97b1ef3b8687f: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:42.811231 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:42.811268 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1f2efe25014e49d8b36ca467c0f40ef3: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server b934fd09590a40ada094b9715d990a00
I20240501 14:09:42.833823 923 rebalancer_tool.cc:1074] tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'b934fd09590a40ada094b9715d990a00' move scheduled
I20240501 14:09:42.856660 923 rebalancer_tool.cc:1074] tablet 0399a419a25c4981a2ee90d76cbf59b1: '6a6fcc099d1340c383e5a429643d5b7c' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move scheduled
I20240501 14:09:42.920876 923 rebalancer_tool.cc:1074] tablet 3c8200d9c20e4140906c71c746badb94: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '9438789dc61f4501ad1d4ceaeccee458' move scheduled
I20240501 14:09:43.103277 923 rebalancer_tool.cc:1074] tablet b67bd0eb45a3486492b75ca3ad3be524: '0e4622d80cc94f12b492bdd7597a61e1' -> 'a0caf2974041408bb32362bfc4284eaa' move scheduled
I20240501 14:09:43.155651 923 rebalancer_tool.cc:1074] tablet a7810938b8fb406e9c6ac438f54afd03: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'b934fd09590a40ada094b9715d990a00' move scheduled
I20240501 14:09:43.261063 923 rebalancer_tool.cc:1074] tablet 5b2e30cb9de14fe19273b3cef801cb67: '6a6fcc099d1340c383e5a429643d5b7c' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move scheduled
I20240501 14:09:43.296597 923 rebalancer_tool.cc:1074] tablet b1006490ec2841b2a3365295caaf19e1: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '9438789dc61f4501ad1d4ceaeccee458' move scheduled
I20240501 14:09:43.338573 923 rebalancer_tool.cc:1074] tablet 5d994081c3414f779ba2cae431cb1d97: '0e4622d80cc94f12b492bdd7597a61e1' -> 'a0caf2974041408bb32362bfc4284eaa' move scheduled
I20240501 14:09:43.395150 923 rebalancer_tool.cc:1074] tablet 2e68be1521964ba8b864801fdf3b0df3: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move scheduled
I20240501 14:09:43.464957 923 rebalancer_tool.cc:1074] tablet 96f5e4e25a674a99a6762757f0ab1545: '6a6fcc099d1340c383e5a429643d5b7c' -> 'b934fd09590a40ada094b9715d990a00' move scheduled
I20240501 14:09:43.552019 923 rebalancer_tool.cc:1084] tablet f8cc4d3985224178aa008b4c9de5625b: 'e7eada5032fb45c3a5e3196e2b3faaab' -> 'a0caf2974041408bb32362bfc4284eaa' move ignored: Illegal state: tablet f8cc4d3985224178aa008b4c9de5625b: replica a0caf2974041408bb32362bfc4284eaa is already present
I20240501 14:09:43.552141 923 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:09:44.110132 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01dc7a764a349c9a4a9180993e4fbc3: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:44.110270 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77c4973914b2412cbc293014310ba9c6: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:44.110337 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:44.110452 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbe6d56762d84933a7b7ddaf8c314b4f: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:44.110512 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd036c60a12148d2ada73beb29aedcbf: could not find any suitable replica to move from server 9438789dc61f4501ad1d4ceaeccee458 to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:44.110584 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 026cd477adec4547bee4fb8004a63348: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:44.110632 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bd1074f581c447ee86e97b1ef3b8687f: could not find any suitable replica to move from server 6a6fcc099d1340c383e5a429643d5b7c to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:44.110684 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1f2efe25014e49d8b36ca467c0f40ef3: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:44.110735 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:44.110783 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77c4973914b2412cbc293014310ba9c6: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server 9438789dc61f4501ad1d4ceaeccee458
I20240501 14:09:44.349928 923 rebalancer_tool.cc:1074] tablet 72ad9c714b2b4fff94063d2838835173: '0e4622d80cc94f12b492bdd7597a61e1' -> 'a0caf2974041408bb32362bfc4284eaa' move scheduled
I20240501 14:09:44.551056 923 rebalancer_tool.cc:1074] tablet c082ccc777ba4f9a937931f707645e54: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '9438789dc61f4501ad1d4ceaeccee458' move scheduled
I20240501 14:09:44.776008 923 rebalancer_tool.cc:1074] tablet 6f8bd03c05d84fdaaf138a6d0608e30e: '6a6fcc099d1340c383e5a429643d5b7c' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move scheduled
I20240501 14:09:44.964582 923 rebalancer_tool.cc:1074] tablet ea66fe3d0da04d18b961927a9c988a20: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'a0caf2974041408bb32362bfc4284eaa' move scheduled
I20240501 14:09:45.280786 923 rebalancer_tool.cc:1074] tablet 8ffdb76b2c5149a18d45bee9581e605a: 'e7eada5032fb45c3a5e3196e2b3faaab' -> 'b934fd09590a40ada094b9715d990a00' move scheduled
I20240501 14:09:45.390288 923 rebalancer_tool.cc:1074] tablet 8f91909d3b1b46e9b858ce5c5b2e7975: '0e4622d80cc94f12b492bdd7597a61e1' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move scheduled
I20240501 14:09:45.442960 923 rebalancer_tool.cc:1084] tablet bca09920a7904362bc546916e3f6f5ac: '30e0b8849e3841ca9ab0de01e719ee4a' -> '9438789dc61f4501ad1d4ceaeccee458' move ignored: Not found: Peer with uuid 30e0b8849e3841ca9ab0de01e719ee4a not found in consensus config
I20240501 14:09:45.443048 923 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:09:46.185390 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 351ff5b1095247928fbb77893616e4a9: could not find any suitable replica to move from server 6a6fcc099d1340c383e5a429643d5b7c to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:46.185505 923 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01dc7a764a349c9a4a9180993e4fbc3: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server 6a6fcc099d1340c383e5a429643d5b7c
I20240501 14:09:46.237520 923 rebalancer_tool.cc:1074] tablet 9d2c173b2c32494eb16e46a51ca36e31: '9438789dc61f4501ad1d4ceaeccee458' -> '0e4622d80cc94f12b492bdd7597a61e1' move scheduled
I20240501 14:09:46.397718 923 rebalancer_tool.cc:1074] tablet 7a542c07a3f841b0a31051a8a37083bf: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'a0caf2974041408bb32362bfc4284eaa' move scheduled
I20240501 14:09:46.536975 923 rebalancer_tool.cc:1074] tablet f26d16499ab54a3e9059f6e4ff041177: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '9438789dc61f4501ad1d4ceaeccee458' move scheduled
I20240501 14:09:46.556036 923 rebalancer_tool.cc:1136] tablet 5b2e30cb9de14fe19273b3cef801cb67: '6a6fcc099d1340c383e5a429643d5b7c' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move completed: OK
I20240501 14:09:46.869002 923 rebalancer_tool.cc:1136] tablet 2e68be1521964ba8b864801fdf3b0df3: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move completed: OK
I20240501 14:09:46.899145 923 rebalancer_tool.cc:1136] tablet b67bd0eb45a3486492b75ca3ad3be524: '0e4622d80cc94f12b492bdd7597a61e1' -> 'a0caf2974041408bb32362bfc4284eaa' move completed: OK
I20240501 14:09:46.928280 923 rebalancer_tool.cc:1136] tablet 3c8200d9c20e4140906c71c746badb94: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '9438789dc61f4501ad1d4ceaeccee458' move completed: OK
I20240501 14:09:46.968060 923 rebalancer_tool.cc:1136] tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'b934fd09590a40ada094b9715d990a00' move completed: OK
I20240501 14:09:47.056769 923 rebalancer_tool.cc:1136] tablet a7810938b8fb406e9c6ac438f54afd03: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'b934fd09590a40ada094b9715d990a00' move completed: OK
I20240501 14:09:47.131332 923 rebalancer_tool.cc:1136] tablet 0399a419a25c4981a2ee90d76cbf59b1: '6a6fcc099d1340c383e5a429643d5b7c' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move completed: OK
I20240501 14:09:47.167415 923 rebalancer_tool.cc:1136] tablet 96f5e4e25a674a99a6762757f0ab1545: '6a6fcc099d1340c383e5a429643d5b7c' -> 'b934fd09590a40ada094b9715d990a00' move completed: OK
I20240501 14:09:47.259685 923 rebalancer_tool.cc:1136] tablet b1006490ec2841b2a3365295caaf19e1: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '9438789dc61f4501ad1d4ceaeccee458' move completed: OK
I20240501 14:09:47.353348 923 rebalancer_tool.cc:1136] tablet 5d994081c3414f779ba2cae431cb1d97: '0e4622d80cc94f12b492bdd7597a61e1' -> 'a0caf2974041408bb32362bfc4284eaa' move completed: OK
I20240501 14:09:47.530993 923 rebalancer_tool.cc:1123] tablet 6f8bd03c05d84fdaaf138a6d0608e30e: '6a6fcc099d1340c383e5a429643d5b7c' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move is abandoned: Illegal state: 6f8bd03c05d84fdaaf138a6d0608e30e: source replica 6a6fcc099d1340c383e5a429643d5b7c does not have REPLACE attribute set
I20240501 14:09:47.549119 923 rebalancer_tool.cc:1136] tablet 8ffdb76b2c5149a18d45bee9581e605a: 'e7eada5032fb45c3a5e3196e2b3faaab' -> 'b934fd09590a40ada094b9715d990a00' move completed: OK
I20240501 14:09:47.595731 923 rebalancer_tool.cc:1136] tablet 8f91909d3b1b46e9b858ce5c5b2e7975: '0e4622d80cc94f12b492bdd7597a61e1' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move completed: OK
I20240501 14:09:47.668032 923 rebalancer_tool.cc:1084] tablet 4603d1b1b1d441b1b0be0464e2f2c9e2: '6a6fcc099d1340c383e5a429643d5b7c' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move ignored: Illegal state: tablet 4603d1b1b1d441b1b0be0464e2f2c9e2: replica a63e0dd1ba2e4a8db8ace9718a700c32 is already present
I20240501 14:09:47.668103 923 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): unacceptable health status UNAVAILABLE
@ 0x7f5a0d646807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5a0de846db start_thread at ??:0
@ 0x7f5a0b3bc71f clone at ??:0
I20240501 14:09:47.940802 31448 raft_consensus.cc:1203] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:09:47.957501 3992 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:47.957803 3992 tablet_bootstrap.cc:492] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: Bootstrap complete.
I20240501 14:09:47.957993 3992 ts_tablet_manager.cc:1367] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent bootstrapping tablet: real 0.064s user 0.030s sys 0.000s
I20240501 14:09:47.958431 3992 raft_consensus.cc:348] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } }
I20240501 14:09:47.958559 3992 raft_consensus.cc:729] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Initialized, Role: LEARNER
I20240501 14:09:47.958663 3992 consensus_queue.cc:260] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true } }
I20240501 14:09:47.958966 3992 ts_tablet_manager.cc:1398] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:47.960048 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session 0e4622d80cc94f12b492bdd7597a61e1-3c8200d9c20e4140906c71c746badb94 received from {username='slave'} at 127.23.191.65:46557
I20240501 14:09:47.960124 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session 0e4622d80cc94f12b492bdd7597a61e1-3c8200d9c20e4140906c71c746badb94 on tablet 3c8200d9c20e4140906c71c746badb94 with peer 0e4622d80cc94f12b492bdd7597a61e1
I20240501 14:09:47.970491 31317 consensus_queue.cc:237] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:47.972585 31448 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:47.972637 734 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:47.973032 970 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:09:47.973176 31256 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.973277 3210 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:09:47.973681 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4603d1b1b1d441b1b0be0464e2f2c9e2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.974957 2556 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:47.975481 734 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:47.977944 31448 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:47.978410 31047 catalog_manager.cc:5484] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 8 to 9, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:47.985872 31581 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "043344ce36a74a18a0d67ff5bd473e68"
dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43516
I20240501 14:09:47.986011 31581 raft_consensus.cc:595] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:47.998713 31189 consensus_queue.cc:237] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } }
W20240501 14:09:48.000909 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): Couldn't send request to peer e7eada5032fb45c3a5e3196e2b3faaab. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.001391 472 raft_consensus.cc:1261] T a7810938b8fb406e9c6ac438f54afd03 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:48.002216 3420 consensus_queue.cc:1035] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 14:09:48.002336 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.005942 1589 raft_consensus.cc:979] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab: : Instructing follower 6a6fcc099d1340c383e5a429643d5b7c to start an election
I20240501 14:09:48.006031 1589 raft_consensus.cc:1067] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Signalling peer 6a6fcc099d1340c383e5a429643d5b7c to start an election
I20240501 14:09:48.006884 31448 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "043344ce36a74a18a0d67ff5bd473e68"
dest_uuid: "6a6fcc099d1340c383e5a429643d5b7c"
from {username='slave'} at 127.23.191.68:50033
I20240501 14:09:48.007004 31448 raft_consensus.cc:482] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:48.007057 31448 raft_consensus.cc:3037] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:48.008060 31448 raft_consensus.cc:504] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:48.008502 31448 leader_election.cc:288] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 2 election: Requested vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
I20240501 14:09:48.009196 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043344ce36a74a18a0d67ff5bd473e68" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:48.009232 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043344ce36a74a18a0d67ff5bd473e68" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:48.009305 31189 raft_consensus.cc:3037] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:48.009387 31581 raft_consensus.cc:3032] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:48.009431 31581 raft_consensus.cc:729] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Running, Role: LEADER
I20240501 14:09:48.009569 31581 consensus_queue.cc:260] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:48.009742 31581 raft_consensus.cc:3037] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:48.010282 31189 raft_consensus.cc:2454] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 2.
I20240501 14:09:48.010782 31581 raft_consensus.cc:2454] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 2.
I20240501 14:09:48.011147 31386 leader_election.cc:302] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, 6a6fcc099d1340c383e5a429643d5b7c, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:48.011335 2175 raft_consensus.cc:2781] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:48.011425 2175 raft_consensus.cc:686] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 2 LEADER]: Becoming Leader. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Running, Role: LEADER
I20240501 14:09:48.011565 2175 consensus_queue.cc:237] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:48.012436 734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043344ce36a74a18a0d67ff5bd473e68" candidate_uuid: "6a6fcc099d1340c383e5a429643d5b7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a0caf2974041408bb32362bfc4284eaa"
I20240501 14:09:48.012567 734 raft_consensus.cc:3037] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:48.013598 734 raft_consensus.cc:2454] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a6fcc099d1340c383e5a429643d5b7c in term 2.
I20240501 14:09:48.014274 31041 catalog_manager.cc:5484] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: term changed from 1 to 2, leader changed from e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) to 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67). New cstate: current_term: 2 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:48.081238 2877 raft_consensus.cc:1050] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER 30e0b8849e3841ca9ab0de01e719ee4a to VOTER
I20240501 14:09:48.081732 2877 consensus_queue.cc:237] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } }
I20240501 14:09:48.083302 734 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:09:48.083453 472 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:09:48.083693 2891 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:48.083894 2453 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:48.084759 31448 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:09:48.085320 31317 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:09:48.085289 2453 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:48.085602 2877 raft_consensus.cc:1013] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: attempt to promote peer a0caf2974041408bb32362bfc4284eaa: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:09:48.085799 2877 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:48.087507 31189 raft_consensus.cc:3037] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEARNER]: Advancing to term 2
I20240501 14:09:48.087652 2453 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.088478 3439 raft_consensus.cc:979] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1: : Instructing follower b934fd09590a40ada094b9715d990a00 to start an election
I20240501 14:09:48.088554 3439 raft_consensus.cc:1067] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Signalling peer b934fd09590a40ada094b9715d990a00 to start an election
I20240501 14:09:48.088608 31189 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 LEARNER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:09:48.089200 472 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.089711 473 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545"
dest_uuid: "b934fd09590a40ada094b9715d990a00"
from {username='slave'} at 127.23.191.65:40699
I20240501 14:09:48.089836 473 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:48.089885 473 raft_consensus.cc:3037] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:48.090101 31317 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.090754 473 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:48.091993 473 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 2 election: Requested vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:09:48.092672 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:48.092783 31189 raft_consensus.cc:3032] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:48.092821 31189 raft_consensus.cc:729] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Running, Role: LEADER
I20240501 14:09:48.092984 31189 consensus_queue.cc:260] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:48.093135 31189 raft_consensus.cc:3037] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:48.093011 31041 catalog_manager.cc:5484] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 12 to 13, 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:48.094004 31189 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
I20240501 14:09:48.094283 734 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.095075 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab"
I20240501 14:09:48.095191 31581 raft_consensus.cc:3037] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:48.096101 31581 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
I20240501 14:09:48.096962 1589 raft_consensus.cc:1050] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER a0caf2974041408bb32362bfc4284eaa to VOTER
I20240501 14:09:48.096992 410 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters:
I20240501 14:09:48.097047 2419 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: NON_VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:48.097219 2530 raft_consensus.cc:2781] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:48.097378 2530 raft_consensus.cc:686] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [term 2 LEADER]: Becoming Leader. State: Replica: b934fd09590a40ada094b9715d990a00, State: Running, Role: LEADER
I20240501 14:09:48.097326 1589 consensus_queue.cc:237] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } }
I20240501 14:09:48.097625 2530 consensus_queue.cc:237] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:48.098429 31448 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.098743 473 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:09:48.099082 2453 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:48.100018 31317 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:09:48.100420 2891 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:48.102085 734 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:09:48.102553 2891 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:48.107443 2419 raft_consensus.cc:1050] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c: attempting to promote NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 to VOTER
W20240501 14:09:48.107522 2419 raft_consensus.cc:1053] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c: Unable to promote non-voter 0e4622d80cc94f12b492bdd7597a61e1: Illegal state: Leader has not yet committed an operation in its own term
I20240501 14:09:48.107686 31042 catalog_manager.cc:5484] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 reported cstate change: term changed from 1 to 2, leader changed from 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) to b934fd09590a40ada094b9715d990a00 (127.23.191.69). New cstate: current_term: 2 leader_uuid: "b934fd09590a40ada094b9715d990a00" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:48.108400 31033 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 72ad9c714b2b4fff94063d2838835173 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER failed with leader e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } }
I20240501 14:09:48.108656 2891 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.110452 734 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.110742 31448 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:09:48.110687 31317 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.111141 1589 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:48.112130 31448 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.114059 31039 catalog_manager.cc:5484] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 13 to 14, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.117595 31581 consensus_queue.cc:237] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } }
I20240501 14:09:48.119119 31317 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 14:09:48.119176 734 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 14:09:48.119752 2891 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 14:09:48.119869 473 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 14:09:48.120232 2891 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 14:09:48.120451 2683 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:48.121096 473 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.124001 1589 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.124707 473 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.125989 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 72ad9c714b2b4fff94063d2838835173 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:48.126150 734 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.127177 31317 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.129508 31040 catalog_manager.cc:5484] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 14 to 15, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.134877 31429 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ad9c714b2b4fff94063d2838835173 with delete_type TABLET_DATA_TOMBSTONED (TS 6a6fcc099d1340c383e5a429643d5b7c not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:41412
I20240501 14:09:48.136873 4122 tablet_replica.cc:330] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c: stopping tablet replica
I20240501 14:09:48.137115 31581 consensus_queue.cc:237] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } }
I20240501 14:09:48.138327 31317 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240501 14:09:48.138541 734 raft_consensus.cc:1261] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240501 14:09:48.138909 2453 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 14:09:48.139075 2891 consensus_queue.cc:1035] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 14:09:48.139683 4122 raft_consensus.cc:2229] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:48.139832 4122 raft_consensus.cc:2258] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:48.140383 31033 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 72ad9c714b2b4fff94063d2838835173 with cas_config_opid_index 13: aborting the task: latest config opid_index 15; task opid_index 13
I20240501 14:09:48.140937 4122 ts_tablet_manager.cc:1875] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:48.140993 2453 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.143687 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 72ad9c714b2b4fff94063d2838835173 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:48.143796 31041 catalog_manager.cc:5484] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 15 to 16, VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:48.144225 31317 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.144302 734 raft_consensus.cc:2932] T 72ad9c714b2b4fff94063d2838835173 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } } }
I20240501 14:09:48.149717 453 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ad9c714b2b4fff94063d2838835173 with delete_type TABLET_DATA_TOMBSTONED (TS b934fd09590a40ada094b9715d990a00 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:48254
I20240501 14:09:48.150665 4122 ts_tablet_manager.cc:1888] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.14
I20240501 14:09:48.151063 4122 log.cc:1196] T 72ad9c714b2b4fff94063d2838835173 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/72ad9c714b2b4fff94063d2838835173
I20240501 14:09:48.151623 31032 catalog_manager.cc:4830] TS 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): tablet 72ad9c714b2b4fff94063d2838835173 (table rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]) successfully deleted
I20240501 14:09:48.151733 4128 tablet_replica.cc:330] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00: stopping tablet replica
I20240501 14:09:48.151862 4128 raft_consensus.cc:2229] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:48.152078 4128 raft_consensus.cc:2258] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:48.153053 4128 ts_tablet_manager.cc:1875] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:48.155234 4128 ts_tablet_manager.cc:1888] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.15
I20240501 14:09:48.155313 4128 log.cc:1196] T 72ad9c714b2b4fff94063d2838835173 P b934fd09590a40ada094b9715d990a00: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-4/wal/wals/72ad9c714b2b4fff94063d2838835173
I20240501 14:09:48.155807 31033 catalog_manager.cc:4830] TS b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): tablet 72ad9c714b2b4fff94063d2838835173 (table rebalance_test_table_4 [id=026cd477adec4547bee4fb8004a63348]) successfully deleted
W20240501 14:09:48.157140 31387 connection.cc:535] client connection to 127.23.191.70:38181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:09:48.157219 2175 raft_consensus.cc:1050] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c: attempting to promote NON_VOTER a0caf2974041408bb32362bfc4284eaa to VOTER
W20240501 14:09:48.157276 31125 connection.cc:535] client connection to 127.23.191.70:38181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:09:48.157609 2175 consensus_queue.cc:237] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
W20240501 14:09:48.158535 31256 connection.cc:535] client connection to 127.23.191.70:38181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:09:48.159411 31317 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 14:09:48.159480 31256 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:48.159502 4110 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:09:48.097400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.191.70:38181 (local address 127.23.191.69:60041)
0501 14:09:48.097609 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0501 14:09:48.097624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 14:09:48.097730 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:09:48.158937 (+ 61207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
W20240501 14:09:48.159525 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:48.159556 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:48.159590 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:48.159690 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:09:48.160410 2419 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:09:48.160543 3877 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:09:48.157924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.191.70:38181 (local address 127.23.191.65:37109)
0501 14:09:48.158154 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0501 14:09:48.158166 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 14:09:48.158255 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:09:48.160105 (+ 1850us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":158}
W20240501 14:09:48.160666 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:48.160709 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:48.160801 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.161026 2419 raft_consensus.cc:1050] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c: attempting to promote NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab to VOTER
I20240501 14:09:48.161293 734 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa [term 1 LEARNER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:48.161398 2419 consensus_queue.cc:237] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false } }
W20240501 14:09:48.161702 3876 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:09:48.157964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.191.70:38181 (local address 127.23.191.67:42685)
0501 14:09:48.160114 (+ 2150us) negotiation.cc:107] Waiting for socket to connect
0501 14:09:48.160125 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 14:09:48.160207 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:09:48.161326 (+ 1119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2106,"thread_start_us":73,"threads_started":1}
I20240501 14:09:48.161666 2175 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:48.162685 31189 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:48.162707 737 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 14:09:48.163161 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.163201 31581 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEARNER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:48.163354 4136 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:48.163755 4137 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240501 14:09:48.164595 2419 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:09:48.165223 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.168577 2419 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false } } }
I20240501 14:09:48.170333 31189 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false } } }
W20240501 14:09:48.171907 411 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:09:48.172451 31581 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false } } }
I20240501 14:09:48.172629 31042 catalog_manager.cc:5484] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 7 to 8, e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false } } }
I20240501 14:09:48.173142 734 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false } } }
I20240501 14:09:48.175043 4137 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:48.177289 734 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:48.178086 31448 consensus_queue.cc:237] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false } }
I20240501 14:09:48.178056 31317 raft_consensus.cc:2932] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:48.179749 31039 catalog_manager.cc:5484] T 7a542c07a3f841b0a31051a8a37083bf P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 8 to 9, a0caf2974041408bb32362bfc4284eaa (127.23.191.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:48.180867 734 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:48.181305 31581 raft_consensus.cc:1261] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:48.181681 2175 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:48.181989 2175 consensus_queue.cc:1035] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:48.183749 970 raft_consensus.cc:1050] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab to VOTER
I20240501 14:09:48.184123 970 consensus_queue.cc:237] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false } }
I20240501 14:09:48.184379 382 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false } } }
I20240501 14:09:48.184623 734 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false } } }
I20240501 14:09:48.185302 31581 raft_consensus.cc:1261] T 8f91909d3b1b46e9b858ce5c5b2e7975 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:48.185645 2556 consensus_queue.cc:1035] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 14:09:48.187384 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.187351 31042 catalog_manager.cc:5484] T ad0dac76fd0c4ae6b8e0549376c98e5d P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 8 to 9, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false } } }
W20240501 14:09:48.187487 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.187654 31032 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ad0dac76fd0c4ae6b8e0549376c98e5d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:48.190348 31581 raft_consensus.cc:2932] T ad0dac76fd0c4ae6b8e0549376c98e5d P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false } } }
I20240501 14:09:48.194244 31169 tablet_service.cc:1514] Processing DeleteTablet for tablet ad0dac76fd0c4ae6b8e0549376c98e5d with delete_type TABLET_DATA_TOMBSTONED (TS 0e4622d80cc94f12b492bdd7597a61e1 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59614
I20240501 14:09:48.195082 4163 tablet_replica.cc:330] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1: stopping tablet replica
I20240501 14:09:48.195333 4163 raft_consensus.cc:2229] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:48.195528 4163 raft_consensus.cc:2258] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:48.196743 4163 ts_tablet_manager.cc:1875] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:48.198953 4163 ts_tablet_manager.cc:1888] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:09:48.199019 4163 log.cc:1196] T ad0dac76fd0c4ae6b8e0549376c98e5d P 0e4622d80cc94f12b492bdd7597a61e1: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-0/wal/wals/ad0dac76fd0c4ae6b8e0549376c98e5d
I20240501 14:09:48.199481 31031 catalog_manager.cc:4830] TS 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841): tablet ad0dac76fd0c4ae6b8e0549376c98e5d (table rebalance_test_table_2 [id=e134e011714149fd80d2248e61c82357]) successfully deleted
I20240501 14:09:48.211427 31189 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b2e30cb9de14fe19273b3cef801cb67"
dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50194
I20240501 14:09:48.211556 31189 raft_consensus.cc:595] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:48.213639 31317 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 2 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:09:48.214176 4137 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:48.217361 734 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 2 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:09:48.218410 4136 raft_consensus.cc:1050] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c: attempting to promote NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 to VOTER
I20240501 14:09:48.218815 4136 consensus_queue.cc:237] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } }
I20240501 14:09:48.220036 31317 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 2 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:48.220348 734 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 2 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:48.220420 2175 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:48.220937 2175 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:48.222517 31189 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 LEARNER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:48.222962 2419 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:48.223093 4137 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.224133 31189 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.224807 31317 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.226262 31039 catalog_manager.cc:5484] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 7 to 9, 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:48.227413 737 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.254911 31189 raft_consensus.cc:1203] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240501 14:09:48.274689 31189 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b2e30cb9de14fe19273b3cef801cb67"
dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50246
I20240501 14:09:48.274828 31189 raft_consensus.cc:595] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:09:48.277263 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:48.308089 31189 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b2e30cb9de14fe19273b3cef801cb67"
dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50296
I20240501 14:09:48.308214 31189 raft_consensus.cc:595] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:09:48.309576 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:48.317406 3420 raft_consensus.cc:979] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1: : Instructing follower b934fd09590a40ada094b9715d990a00 to start an election
I20240501 14:09:48.317515 3420 raft_consensus.cc:1067] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Signalling peer b934fd09590a40ada094b9715d990a00 to start an election
I20240501 14:09:48.317934 473 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b2e30cb9de14fe19273b3cef801cb67"
dest_uuid: "b934fd09590a40ada094b9715d990a00"
from {username='slave'} at 127.23.191.65:40699
I20240501 14:09:48.318063 473 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:48.318130 473 raft_consensus.cc:3037] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:48.319052 473 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:48.319571 473 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 2 election: Requested vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
I20240501 14:09:48.319954 31189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e30cb9de14fe19273b3cef801cb67" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: true dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1"
I20240501 14:09:48.320067 31189 raft_consensus.cc:3032] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:48.320109 31189 raft_consensus.cc:729] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0e4622d80cc94f12b492bdd7597a61e1, State: Running, Role: LEADER
I20240501 14:09:48.320238 31189 consensus_queue.cc:260] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:48.320425 31189 raft_consensus.cc:3037] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:09:48.320958 411 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:48.321386 31189 raft_consensus.cc:2454] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
I20240501 14:09:48.323215 737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e30cb9de14fe19273b3cef801cb67" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: true dest_uuid: "a0caf2974041408bb32362bfc4284eaa"
I20240501 14:09:48.323330 737 raft_consensus.cc:3037] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:48.324029 737 raft_consensus.cc:2454] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
I20240501 14:09:48.324321 409 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, a0caf2974041408bb32362bfc4284eaa, b934fd09590a40ada094b9715d990a00; no voters: a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:48.324532 2530 raft_consensus.cc:2781] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:48.324620 2530 raft_consensus.cc:686] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [term 2 LEADER]: Becoming Leader. State: Replica: b934fd09590a40ada094b9715d990a00, State: Running, Role: LEADER
I20240501 14:09:48.324756 2530 consensus_queue.cc:237] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:48.326407 31042 catalog_manager.cc:5484] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 reported cstate change: term changed from 1 to 2, leader changed from 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) to b934fd09590a40ada094b9715d990a00 (127.23.191.69). New cstate: current_term: 2 leader_uuid: "b934fd09590a40ada094b9715d990a00" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:48.376014 3420 raft_consensus.cc:1050] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c to VOTER
I20240501 14:09:48.376484 3420 consensus_queue.cc:237] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } }
I20240501 14:09:48.377797 31448 raft_consensus.cc:1261] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:48.378427 31317 raft_consensus.cc:1261] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:48.379038 31448 raft_consensus.cc:1203] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240501 14:09:48.379559 31125 consensus_peers.cc:485] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.379841 3439 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:48.380110 3439 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:48.381767 4318 ts_tablet_manager.cc:902] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:09:48.382313 4318 tablet_copy_client.cc:308] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:09:48.382860 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 from peer b934fd09590a40ada094b9715d990a00 ({username='slave'} at 127.23.191.69:54165)
I20240501 14:09:48.382957 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 from peer b934fd09590a40ada094b9715d990a00 at {username='slave'} at 127.23.191.69:54165: session id = b934fd09590a40ada094b9715d990a00-4603d1b1b1d441b1b0be0464e2f2c9e2
I20240501 14:09:48.383920 31340 tablet_copy_source_session.cc:205] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:48.384007 3420 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:09:48.384433 31317 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:09:48.384919 4318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.387179 4318 tablet_copy_client.cc:783] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:48.387319 4318 tablet_copy_client.cc:647] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:48.387668 31448 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:09:48.387650 31040 catalog_manager.cc:5484] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 7 to 8, 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
W20240501 14:09:48.389957 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:09:48.390784 4318 tablet_copy_client.cc:523] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:48.390980 31189 consensus_queue.cc:237] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } }
I20240501 14:09:48.392006 4318 tablet_bootstrap.cc:492] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00: Bootstrap starting.
I20240501 14:09:48.392012 31448 raft_consensus.cc:1261] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:48.392629 3439 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:48.393625 31317 raft_consensus.cc:1261] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:48.394038 3439 consensus_queue.cc:1035] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:48.394577 3439 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:09:48.396067 31448 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:09:48.396136 31317 raft_consensus.cc:2932] T 6f8bd03c05d84fdaaf138a6d0608e30e P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:09:48.396679 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6f8bd03c05d84fdaaf138a6d0608e30e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:48.397521 31042 catalog_manager.cc:5484] T 6f8bd03c05d84fdaaf138a6d0608e30e P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 8 to 9, VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:09:48.401582 31034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:48.402390 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6f8bd03c05d84fdaaf138a6d0608e30e: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:48.412886 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:09:48.414170 4318 tablet_bootstrap.cc:492] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:48.414426 4318 tablet_bootstrap.cc:492] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00: Bootstrap complete.
I20240501 14:09:48.414572 4318 ts_tablet_manager.cc:1367] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.004s
I20240501 14:09:48.414929 4318 raft_consensus.cc:348] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:48.415036 4318 raft_consensus.cc:729] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b934fd09590a40ada094b9715d990a00, State: Initialized, Role: LEARNER
I20240501 14:09:48.415122 4318 consensus_queue.cc:260] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:48.415378 4318 ts_tablet_manager.cc:1398] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:48.416015 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session b934fd09590a40ada094b9715d990a00-4603d1b1b1d441b1b0be0464e2f2c9e2 received from {username='slave'} at 127.23.191.69:54165
I20240501 14:09:48.416100 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session b934fd09590a40ada094b9715d990a00-4603d1b1b1d441b1b0be0464e2f2c9e2 on tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 with peer b934fd09590a40ada094b9715d990a00
W20240501 14:09:48.454341 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:48.505209 31256 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.512519 31581 raft_consensus.cc:1261] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:48.513155 382 consensus_queue.cc:1035] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:09:48.517099 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:48.543336 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.581494 31189 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Refusing update from remote peer b934fd09590a40ada094b9715d990a00: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:48.581984 2530 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:48.600757 737 raft_consensus.cc:1261] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa [term 2 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:48.601152 382 consensus_queue.cc:1035] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:48.604552 31448 consensus_queue.cc:237] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } }
I20240501 14:09:48.606333 31189 raft_consensus.cc:1261] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:48.606516 737 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 2 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:48.606984 2419 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:48.607347 2175 consensus_queue.cc:1035] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:48.609570 31189 raft_consensus.cc:1261] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:48.609884 2419 consensus_queue.cc:1035] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:48.611304 4137 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.612146 31189 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.612852 31032 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ea66fe3d0da04d18b961927a9c988a20 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:48.613301 737 raft_consensus.cc:2932] T ea66fe3d0da04d18b961927a9c988a20 P a0caf2974041408bb32362bfc4284eaa [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.614482 31041 catalog_manager.cc:5484] T ea66fe3d0da04d18b961927a9c988a20 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 9 to 10, VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) evicted. New cstate: current_term: 2 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.617038 31298 tablet_service.cc:1514] Processing DeleteTablet for tablet ea66fe3d0da04d18b961927a9c988a20 with delete_type TABLET_DATA_TOMBSTONED (TS 30e0b8849e3841ca9ab0de01e719ee4a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35058
I20240501 14:09:48.617372 4422 tablet_replica.cc:330] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a: stopping tablet replica
I20240501 14:09:48.617503 4422 raft_consensus.cc:2229] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:48.617666 4422 raft_consensus.cc:2258] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:48.618312 4422 ts_tablet_manager.cc:1875] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:48.620451 4422 ts_tablet_manager.cc:1888] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 14:09:48.620529 4422 log.cc:1196] T ea66fe3d0da04d18b961927a9c988a20 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-1/wal/wals/ea66fe3d0da04d18b961927a9c988a20
I20240501 14:09:48.621017 31034 catalog_manager.cc:4830] TS 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265): tablet ea66fe3d0da04d18b961927a9c988a20 (table rebalance_test_table_1 [id=bd1074f581c447ee86e97b1ef3b8687f]) successfully deleted
I20240501 14:09:48.622881 4428 ts_tablet_manager.cc:902] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:09:48.623363 4428 tablet_copy_client.cc:308] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
I20240501 14:09:48.623771 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet a7810938b8fb406e9c6ac438f54afd03 from peer e7eada5032fb45c3a5e3196e2b3faaab ({username='slave'} at 127.23.191.68:47289)
I20240501 14:09:48.623859 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet a7810938b8fb406e9c6ac438f54afd03 from peer e7eada5032fb45c3a5e3196e2b3faaab at {username='slave'} at 127.23.191.68:47289: session id = e7eada5032fb45c3a5e3196e2b3faaab-a7810938b8fb406e9c6ac438f54afd03
I20240501 14:09:48.624732 31209 tablet_copy_source_session.cc:205] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:48.625435 4428 ts_tablet_manager.cc:1875] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:09:48.627251 4428 ts_tablet_manager.cc:1888] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240501 14:09:48.627398 4428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7810938b8fb406e9c6ac438f54afd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.628259 4428 tablet_copy_client.cc:783] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:48.628510 4428 tablet_copy_client.cc:647] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:48.630065 4428 tablet_copy_client.cc:523] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:48.631099 4428 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:09:48.650429 4428 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:48.650717 4428 tablet_bootstrap.cc:492] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap complete.
I20240501 14:09:48.650867 4428 ts_tablet_manager.cc:1367] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20240501 14:09:48.651207 4428 raft_consensus.cc:348] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } }
I20240501 14:09:48.651402 4428 raft_consensus.cc:729] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: LEARNER
I20240501 14:09:48.651500 4428 consensus_queue.cc:260] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } }
I20240501 14:09:48.651747 4428 ts_tablet_manager.cc:1398] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:48.652302 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session e7eada5032fb45c3a5e3196e2b3faaab-a7810938b8fb406e9c6ac438f54afd03 received from {username='slave'} at 127.23.191.68:47289
I20240501 14:09:48.652385 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session e7eada5032fb45c3a5e3196e2b3faaab-a7810938b8fb406e9c6ac438f54afd03 on tablet a7810938b8fb406e9c6ac438f54afd03 with peer e7eada5032fb45c3a5e3196e2b3faaab
I20240501 14:09:48.702764 31581 raft_consensus.cc:1261] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Refusing update from remote peer b934fd09590a40ada094b9715d990a00: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:48.703212 2530 consensus_queue.cc:1035] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:48.757148 3258 raft_consensus.cc:1050] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER 0e4622d80cc94f12b492bdd7597a61e1 to VOTER
I20240501 14:09:48.757591 3258 consensus_queue.cc:237] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } }
I20240501 14:09:48.758963 473 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:09:48.759030 31189 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:09:48.759385 2513 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:48.759560 2556 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 14:09:48.760556 31256 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.762100 2513 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.762670 31189 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.764020 473 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.765368 31040 catalog_manager.cc:5484] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 12 to 13, 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.769037 31317 consensus_queue.cc:237] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } }
I20240501 14:09:48.770107 473 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:09:48.770517 4468 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:48.770591 31189 raft_consensus.cc:1261] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:09:48.770961 4468 consensus_queue.cc:1035] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:48.772816 3258 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.773233 31189 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.775092 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3c8200d9c20e4140906c71c746badb94 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:48.776083 473 raft_consensus.cc:2932] T 3c8200d9c20e4140906c71c746badb94 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.776243 31041 catalog_manager.cc:5484] T 3c8200d9c20e4140906c71c746badb94 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 13 to 14, VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { promote: false } } }
I20240501 14:09:48.782856 31189 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Refusing update from remote peer b934fd09590a40ada094b9715d990a00: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240501 14:09:48.783564 4109 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240501 14:09:48.783602 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:48.784480 473 raft_consensus.cc:1203] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240501 14:09:48.876482 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:48.894999 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:48.905019 737 raft_consensus.cc:1261] T 5b2e30cb9de14fe19273b3cef801cb67 P a0caf2974041408bb32362bfc4284eaa [term 2 FOLLOWER]: Refusing update from remote peer b934fd09590a40ada094b9715d990a00: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240501 14:09:48.905412 4109 consensus_queue.cc:1035] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240501 14:09:48.909341 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.955302 4518 raft_consensus.cc:1050] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab: attempting to promote NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c to VOTER
I20240501 14:09:48.955719 4518 consensus_queue.cc:237] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } }
I20240501 14:09:48.957020 31189 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:48.957020 31448 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:48.957396 4520 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:09:48.957522 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.957558 4521 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:48.959262 4520 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:09:48.959715 31448 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:09:48.961649 31189 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:09:48.962239 31042 catalog_manager.cc:5484] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 7 to 8, 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
W20240501 14:09:49.005199 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:09:49.181048 31581 raft_consensus.cc:1203] T a7810938b8fb406e9c6ac438f54afd03 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240501 14:09:49.207968 31448 consensus_queue.cc:237] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:09:49.209023 31189 raft_consensus.cc:1261] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:49.209084 737 raft_consensus.cc:1261] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa [term 2 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:49.209403 4593 consensus_queue.cc:1035] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:49.209590 4592 consensus_queue.cc:1035] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:49.211088 2419 raft_consensus.cc:2932] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:49.211530 31189 raft_consensus.cc:2932] T 043344ce36a74a18a0d67ff5bd473e68 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:49.211555 737 raft_consensus.cc:2932] T 043344ce36a74a18a0d67ff5bd473e68 P a0caf2974041408bb32362bfc4284eaa [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:09:49.212991 31032 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 043344ce36a74a18a0d67ff5bd473e68 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:49.213971 31041 catalog_manager.cc:5484] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 5 to 7, VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) evicted. New cstate: current_term: 2 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:49.216236 31561 tablet_service.cc:1514] Processing DeleteTablet for tablet 043344ce36a74a18a0d67ff5bd473e68 with delete_type TABLET_DATA_TOMBSTONED (TS e7eada5032fb45c3a5e3196e2b3faaab not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34312
I20240501 14:09:49.216575 4594 tablet_replica.cc:330] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab: stopping tablet replica
I20240501 14:09:49.216697 4594 raft_consensus.cc:2229] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:49.216786 4594 raft_consensus.cc:2258] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:49.217384 4594 ts_tablet_manager.cc:1875] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:09:49.220006 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:49.220136 4594 ts_tablet_manager.cc:1888] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:49.220211 4594 log.cc:1196] T 043344ce36a74a18a0d67ff5bd473e68 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-3/wal/wals/043344ce36a74a18a0d67ff5bd473e68
I20240501 14:09:49.220788 31033 catalog_manager.cc:4830] TS e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): tablet 043344ce36a74a18a0d67ff5bd473e68 (table rebalance_test_table_7 [id=855c959aa35441b989d386caf8f8eded]) successfully deleted
I20240501 14:09:49.228453 4518 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:49.228550 4518 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:49.229280 4518 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:09:49.230389 473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:09:49.230530 473 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:09:49.230638 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:49.231020 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:09:49.231087 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:49.231297 4518 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:49.270128 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:09:49.275308 2513 raft_consensus.cc:1050] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER b934fd09590a40ada094b9715d990a00 to VOTER
I20240501 14:09:49.275768 2513 consensus_queue.cc:237] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:49.277350 737 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:49.277659 31448 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:49.277797 473 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00 [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:49.277781 4656 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:49.278152 4657 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:09:49.278345 31256 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:49.278450 31256 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:49.278591 4657 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:49.280148 4656 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:49.280567 31448 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:49.282766 473 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:49.283013 31040 catalog_manager.cc:5484] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 9 to 10, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:49.283994 737 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b934fd09590a40ada094b9715d990a00 (127.23.191.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: NON_VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: true replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:49.288307 31317 consensus_queue.cc:237] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }
I20240501 14:09:49.289414 473 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:49.290149 4658 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 14:09:49.290381 31256 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:49.290827 737 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:49.290897 31448 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:09:49.291174 2513 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:49.291394 4658 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:49.293097 4658 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:49.293483 31448 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:49.293535 737 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:49.295326 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:49.295243 473 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } } }
I20240501 14:09:49.296226 31042 catalog_manager.cc:5484] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 10 to 11, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:49.299216 31317 consensus_queue.cc:237] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:49.300308 737 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:49.300674 473 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:49.300668 4656 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:49.301582 4656 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:49.301743 31448 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:09:49.302186 4657 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:49.303174 4657 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:49.303486 473 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:49.304322 31448 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:49.304541 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:49.306684 31039 catalog_manager.cc:5484] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 11 to 12, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:49.307134 737 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:49.309991 31317 consensus_queue.cc:237] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:49.311345 737 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:09:49.311394 473 raft_consensus.cc:1261] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:09:49.311698 4468 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:49.311877 4656 consensus_queue.cc:1035] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:49.313577 4468 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:49.313872 473 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:49.313872 737 raft_consensus.cc:2932] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:49.315209 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:49.315953 31041 catalog_manager.cc:5484] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 12 to 13, VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:09:49.318054 31429 tablet_service.cc:1514] Processing DeleteTablet for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 with delete_type TABLET_DATA_TOMBSTONED (TS 6a6fcc099d1340c383e5a429643d5b7c not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:41412
I20240501 14:09:49.318468 4678 tablet_replica.cc:330] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c: stopping tablet replica
I20240501 14:09:49.318605 4678 raft_consensus.cc:2229] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:49.319068 4678 raft_consensus.cc:2258] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:49.319905 4678 ts_tablet_manager.cc:1875] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:49.321715 4678 ts_tablet_manager.cc:1888] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 14:09:49.321779 4678 log.cc:1196] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 6a6fcc099d1340c383e5a429643d5b7c: Deleting WAL directory at /tmp/dist-test-tasklfWEhd/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572530256240-24317-0/raft_consensus-itest-cluster/ts-2/wal/wals/4603d1b1b1d441b1b0be0464e2f2c9e2
I20240501 14:09:49.322227 31032 catalog_manager.cc:4830] TS 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): tablet 4603d1b1b1d441b1b0be0464e2f2c9e2 (table rebalance_test_table_9 [id=77c4973914b2412cbc293014310ba9c6]) successfully deleted
W20240501 14:09:49.394881 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:49.403362 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:49.462426 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:49.668054 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:49.713435 4329 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:49.854360 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:49.885303 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:49.896071 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:49.897948 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:49.936591 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:50.040226 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:50.127306 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:50.183159 4804 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:50.215174 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:50.256624 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:50.314033 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:50.316227 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:50.345396 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:50.354635 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:50.376174 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:50.466809 31034 catalog_manager.cc:4573] TS 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:50.530815 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:50.549026 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:50.572499 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:50.644454 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:50.694005 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:50.721560 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:50.738165 4329 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:50.790885 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:50.807786 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:50.861937 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:50.901319 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:50.922700 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:50.949095 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:51.124689 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:51.149583 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:51.156729 4446 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:51.156877 4446 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:51.157318 4446 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:09:51.157778 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:51.157928 31580 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:09:51.158388 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:51.158475 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:09:51.158520 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:51.158638 4446 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:51.337023 4804 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:51.364734 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:51.372169 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:51.445932 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:51.456252 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:51.489168 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:51.497749 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4603d1b1b1d441b1b0be0464e2f2c9e2: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:51.536509 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:51.645606 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:51.806527 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:51.924878 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:09:51.948035 4804 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:52.073858 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:52.216681 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:52.232633 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:52.333819 5309 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:52.359508 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:52.503165 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:52.550201 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:52.629774 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:52.651077 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:52.668380 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:52.694448 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:52.723733 4804 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:52.737934 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6f8bd03c05d84fdaaf138a6d0608e30e: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:52.739259 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240501 14:09:52.786073 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:09:52.786721 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:52.829910 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 14:09:52.831600 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:52.834419 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 14:09:52.928355 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:52.984032 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:53.001040 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:53.056003 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:53.069501 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:53.103278 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:53.131958 4446 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:53.132097 4446 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:53.132534 4446 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:09:53.132860 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:09:53.133010 31580 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:09:53.133666 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:53.133782 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:09:53.133831 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:53.133951 4446 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:53.149554 5309 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:53.158043 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:53.241757 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:53.247097 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:53.306751 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:53.339478 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:53.354704 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:53.397320 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:53.419153 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:53.470108 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:53.483059 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:53.544436 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240501 14:09:53.565554 31034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240501 14:09:53.600939 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:53.608196 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:53.764686 5735 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:53.889367 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:53.902729 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:53.913038 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:54.021054 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:54.078804 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:54.100818 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:54.109412 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:54.200321 5735 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:54.370405 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:54.459177 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:54.625947 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:54.627427 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:54.794464 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:54.794514 5968 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:54.971333 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:54.990312 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:55.015004 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:55.021708 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:55.122293 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:55.158900 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:55.223865 6070 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:09:55.306035 5816 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.306178 5816 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:55.306608 5816 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:09:55.307078 473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:09:55.307226 473 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:09:55.307804 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:55.307907 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:55.307951 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:55.308086 5816 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:55.345016 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:55.391130 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:55.410001 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:55.413503 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:55.459847 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:55.460593 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:55.507905 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:55.563835 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:55.606520 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:55.608354 5968 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:55.690536 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:55.692802 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:55.809007 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:55.896281 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:55.907056 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:55.935431 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:55.969404 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:55.982585 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:55.996423 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:56.067565 6070 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:56.069293 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:56.197532 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:56.214097 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:56.263321 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:56.286038 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:56.404443 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:56.438164 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:56.441793 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:56.600288 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:56.665336 6409 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:56.844095 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:56.878302 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:56.878305 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6f8bd03c05d84fdaaf138a6d0608e30e: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:56.906886 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:57.051106 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:57.146060 6409 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:57.262347 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:57.331049 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:57.400030 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:57.400306 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:57.531955 6409 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:57.582084 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:57.660208 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:57.671028 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:57.673077 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:57.789734 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:09:57.822567 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:09:57.840030 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:09:57.840651 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:57.883186 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:09:57.883823 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:57.896409 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:57.898959 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:57.913789 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:58.006181 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:58.008977 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:58.079308 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:58.097904 6730 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:58.127650 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:58.179466 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:58.191217 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:58.223938 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:58.313529 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:58.407325 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:58.408597 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:58.415513 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:58.498136 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:58.582893 6730 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:58.585315 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:58.586544 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:58.596606 5816 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:58.596705 5816 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:09:58.597230 5816 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:09:58.597815 473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:09:58.597940 473 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:09:58.598695 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:58.598810 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:09:58.598865 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:58.599000 6840 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:58.606583 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:58.709858 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:58.711700 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:09:58.779358 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:58.841558 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:58.884814 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:59.004618 6730 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:59.097724 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:59.123942 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:59.214598 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:59.358042 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:59.416616 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:09:59.493813 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:09:59.614642 6730 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:09:59.792224 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:09:59.914132 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:09:59.924978 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:09:59.940059 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:09:59.993942 6730 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:00.006973 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:00.013964 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:00.085783 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:00.194864 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:00.221781 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:00.290920 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:00.310523 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:00.344041 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:00.344538 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:00.402570 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:00.483184 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:00.496456 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:00.550509 6730 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:00.565465 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:00.663106 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:00.683269 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:00.701206 6840 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:00.701345 6840 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:10:00.701766 6840 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:10:00.702160 473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:10:00.702312 473 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:10:00.702808 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:00.702922 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:00.702961 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:10:00.703071 6840 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:00.793862 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:00.853076 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:00.854619 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:00.973850 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:01.043139 7292 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:01.050593 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:01.061612 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:01.076946 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:01.118609 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:01.139655 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:01.154112 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:01.193663 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:01.373970 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:01.445930 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:01.470554 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:01.496001 6730 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:01.505345 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:01.771870 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:01.776969 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:01.831810 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:01.882959 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:01.924728 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:01.965312 7292 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:02.329008 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:02.332285 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:02.362016 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:02.374886 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:02.391268 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:02.401849 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:02.516112 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:02.536384 6730 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:02.690923 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:02.775288 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:02.827919 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:02.870468 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:02.871116 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:02.872370 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:02.924908 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:10:02.925115 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:10:02.925551 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:02.925736 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:02.946169 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:10:02.947031 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:02.956137 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:02.975549 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:10:02.976153 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:03.004420 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:03.036029 7756 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:03.037300 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:03.319662 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:03.355999 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:03.423336 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:03.500357 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:03.506748 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:03.540063 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:03.602267 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:03.620445 7292 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:03.739940 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:03.785303 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:03.833012 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:10:03.833765 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:03.834295 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:03.912515 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:03.939828 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:04.020442 7292 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:04.028082 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:04.252909 4446 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:04.253044 4446 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:10:04.253605 4446 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:10:04.253965 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:10:04.254122 31581 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:10:04.254637 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:04.254748 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:04.254805 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:10:04.254925 4446 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:04.296252 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:04.311093 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:04.412210 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:04.513608 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:04.582041 7292 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:04.625023 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:04.648603 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:04.678392 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:04.734242 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:04.797996 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:04.830142 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:05.072147 7292 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:05.074530 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:05.100764 31034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240501 14:10:05.101471 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6f8bd03c05d84fdaaf138a6d0608e30e: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:05.137709 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:05.140781 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:05.247119 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:05.286660 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:05.290733 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:05.333329 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:05.338059 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:05.342473 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:05.405838 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:05.413126 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:05.449620 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:05.524667 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:05.531769 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:05.552788 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:05.658864 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:05.679752 7292 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:05.793874 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:05.797945 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:05.844861 8427 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:05.844985 8427 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:10:05.845430 8427 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:10:05.845732 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:10:05.845890 31580 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:10:05.846457 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:05.846555 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:05.846607 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:10:05.846736 8427 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:05.847337 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:05.849872 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:05.866566 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:06.032627 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:06.157222 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:06.163300 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:06.279486 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:06.293785 8478 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:06.306331 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:06.350275 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:06.371431 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:06.501621 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:06.727026 8478 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:06.872347 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:06.887801 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:06.931802 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:07.104967 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:07.106142 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:07.196745 8478 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:07.302042 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:07.420687 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:07.434072 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:07.443677 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:07.462940 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:07.518910 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:07.592293 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:07.621016 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:07.730525 8478 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:07.826989 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:07.827301 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:07.845966 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:07.857177 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:07.864153 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:07.901911 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:07.945112 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:07.952937 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:10:07.969926 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:10:07.999644 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:10:08.000276 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:08.009637 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:10:08.010272 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:08.038815 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:08.046243 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:08.084102 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:08.151245 8939 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:08.156519 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:08.177860 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:08.253939 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:08.350258 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:08.448729 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:08.522269 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:08.601374 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:08.698455 9185 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:08.789326 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:08.798789 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:08.828392 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:09.038547 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:09.048225 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:09.128777 9185 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:09.134150 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:09.258143 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:09.269302 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 25 similar messages]
W20240501 14:10:09.288969 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:09.375679 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:09.535676 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:09.605649 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:09.636720 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:09.672058 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:09.679361 9185 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:09.807583 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:09.838675 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:09.885084 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:09.921001 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:10.098984 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:10.161729 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:10.210685 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:10:10.228997 9185 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:10.266233 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:10.288594 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:10.328413 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:10.342783 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:10.365391 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:10.471587 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:10.574999 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:10.578971 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:10.614451 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:10.635609 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:10:10.665836 9531 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:10.730935 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:10.760802 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:10.762758 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:10.825588 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:10.861186 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:10.942031 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:11.019196 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:11.040575 8427 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:11.040707 8427 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:10:11.041204 8427 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:10:11.041463 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:10:11.041625 31580 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:10:11.042341 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:11.042451 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:11.042498 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:10:11.042637 8427 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:11.186686 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:11.209308 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:11.252508 9758 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:11.277109 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:11.459692 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:11.551029 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:11.568504 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:10:11.739766 9758 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:11.767910 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:11.772789 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:11.837916 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:12.002794 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:12.027544 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:12.079936 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:12.198947 9758 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:12.219664 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:12.319617 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:12.374601 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:12.383430 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:12.397059 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:12.439898 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:12.566183 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:12.619180 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:10:12.624548 9758 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:12.725610 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:12.812985 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:12.870005 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:12.874519 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:12.877882 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:12.951483 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:12.997364 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 14:10:12.997999 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:13.029793 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:13.036669 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:10:13.037147 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:13.037714 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:13.046113 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:10:13.068336 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 14:10:13.110275 9758 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:13.111764 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:13.158555 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:13.270653 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:13.319801 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:13.432884 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:13.465545 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:13.489595 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:13.553637 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:13.572103 9185 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:13.634258 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:13.719859 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:13.808722 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:13.967162 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:14.000938 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:14.021359 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:14.099835 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:14.186977 9185 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:14.338956 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:14.375017 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:14.451905 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:14.461534 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 14:10:14.462066 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:14.547914 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:14.550647 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:14.580263 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:14.734047 9185 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:14.753068 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:14.888641 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:14.911425 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:14.972254 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:14.991910 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:15.018786 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:15.025816 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:15.187007 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:15.219892 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:15.248180 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:15.275074 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:10:15.327620 10567 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:15.351222 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:15.353439 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:15.372949 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:15.376981 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:15.383589 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:15.485718 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:15.488768 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:15.528746 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:15.606832 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:15.710222 8427 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:15.710361 8427 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:10:15.710819 8427 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:10:15.711239 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:10:15.711395 31580 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:10:15.711915 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:15.712018 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:15.712066 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:10:15.712198 8427 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:15.777007 10567 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:15.812544 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:15.817678 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:15.943060 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:15.977562 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:16.022970 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:16.052475 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:16.072649 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:16.237610 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:16.245674 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:16.261207 9185 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:16.519800 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:16.535753 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:16.606621 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:16.696398 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:16.731426 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:10:16.764807 9185 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:16.778148 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:16.796422 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:17.020586 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:17.093183 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:17.156688 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:17.235700 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:17.322971 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:10:17.354204 11155 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:17.460453 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:17.490942 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:17.560881 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:17.615623 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:17.634843 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:17.729993 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:17.753890 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:10:17.761487 11155 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:17.771817 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:17.820466 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:17.827699 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:17.905447 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:17.913316 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:17.917443 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:17.939018 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:17.985831 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:18.013935 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:18.032711 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:18.041008 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 14:10:18.041595 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:18.141183 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 14:10:18.232673 9185 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:18.283720 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:18.310081 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:18.312160 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:10:18.400243 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:18.422513 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:18.491402 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:18.576519 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:18.582079 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:18.713205 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:18.723721 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:18.784757 9185 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:18.914932 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:18.963821 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:18.983517 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:19.028385 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:19.183640 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:10:19.184123 10765 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:19.184231 10765 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:10:19.184688 10765 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:10:19.185038 472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:10:19.185178 472 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:10:19.185675 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:19.185761 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:19.185820 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:10:19.185947 10765 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:19.211211 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:19.241423 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:10:19.329340 11155 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:19.370539 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:19.468529 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:19.496752 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:19.580380 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 14:10:19.583397 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:19.619519 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:19.681900 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:10:19.834054 9185 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:19.871932 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:19.902263 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:19.972095 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:20.098906 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:20.122339 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:20.202698 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:20.203924 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:20.213059 11155 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:20.281334 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:20.283351 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:20.330775 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:20.389410 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:20.407876 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:20.506668 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:20.568377 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:20.584262 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:20.610460 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:20.626986 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:20.684939 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:20.735136 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:20.736074 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:20.767882 11949 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:21.000346 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:21.048782 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:21.107430 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:21.246322 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:21.246836 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:21.279003 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:21.311118 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:21.313618 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:21.326622 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:10:21.366566 11949 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:21.415279 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:21.520990 31034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:10:21.521616 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6f8bd03c05d84fdaaf138a6d0608e30e: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:21.678216 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:21.844873 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:21.880349 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:10:21.903064 12222 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:21.907548 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:21.915908 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:21.922060 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:22.234773 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:22.237682 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:22.241151 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:22.359083 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:22.368716 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:10:22.369411 11949 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:22.430609 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:22.489711 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:22.520475 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:22.664484 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:22.760879 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:22.784679 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:22.828799 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:22.833657 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:22.836449 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:10:22.927853 12441 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:22.936980 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:22.971144 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:23.029369 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:23.052175 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:23.069430 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:10:23.070073 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:23.083263 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:23.163012 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:23.194751 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:23.237807 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:10:23.248636 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:23.361200 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:10:23.404304 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:10:23.536325 12441 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:23.543594 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:23.556669 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:23.634102 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:23.732403 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:23.759279 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:23.784569 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:23.923856 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:23.929486 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:10:23.955066 12593 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:23.991629 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:24.085203 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:24.106000 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:24.183954 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:24.260099 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:24.336407 12593 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:24.402905 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:24.515810 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:24.522620 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:24.576330 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:24.641227 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:24.645820 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:24.672214 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:10:24.711674 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:24.734450 12441 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:24.738296 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:24.918020 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:25.019121 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:25.048125 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:25.077634 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:25.139796 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:25.163374 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:25.189940 12441 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:25.191907 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:25.200004 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:25.370160 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:25.467371 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:25.558198 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:25.564532 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:25.630652 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:25.660010 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:25.701813 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:10:25.709815 12441 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:25.771235 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:25.788118 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:25.800961 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:26.059733 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:26.098371 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:26.107391 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:26.130743 13099 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:26.175998 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:26.187628 12886 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:26.187780 12886 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:10:26.188294 12886 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:10:26.188721 473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:10:26.188899 473 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:10:26.189795 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:26.189880 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:26.189921 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:10:26.190047 12886 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:26.221983 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:26.336211 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:26.369313 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:26.508559 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:10:26.565480 12441 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:26.599860 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:26.637583 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:26.681543 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:26.758265 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:26.768054 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:26.866034 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:26.904578 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:26.996206 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:10:27.018395 13302 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:27.106299 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:27.152402 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:27.191478 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:27.208917 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:27.215488 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:27.292668 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:27.331007 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:10:27.534124 13302 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:27.589910 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:27.650182 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:27.685230 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:27.725592 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:27.763172 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:10:27.776283 12886 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:27.776412 12886 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:10:27.776908 12886 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:10:27.777242 473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:10:27.777397 473 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:10:27.778234 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:27.778339 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:27.778381 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:10:27.778497 12886 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:27.814210 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:27.893182 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:10:28.014415 13302 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:28.046025 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:28.065410 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:28.115342 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:28.120268 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:10:28.120926 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:28.137503 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:28.186604 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:28.255926 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:28.267467 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:28.280757 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:10:28.325654 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:28.388051 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240501 14:10:28.467228 13505 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:28.469151 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:10:28.547911 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:28.567955 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:28.594796 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:28.738819 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:28.796698 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:28.804674 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:28.855840 13302 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:28.856406 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:29.017736 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:29.032685 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:29.099267 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:29.271245 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:29.361037 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:29.386343 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:29.397370 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:29.411301 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:10:29.413456 13302 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:29.494678 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:29.561900 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:29.569625 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:29.573493 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:29.667148 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:29.774897 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:29.791824 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:29.910954 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 14:10:29.924243 13302 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:29.994786 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:30.007504 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:30.089474 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:30.189996 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:30.191174 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:30.206020 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:30.270700 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:30.414331 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:30.437966 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:10:30.446125 14186 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:30.451640 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:30.473057 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:30.536978 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:30.708472 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:30.745443 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:30.786039 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:30.791925 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:30.806574 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:30.915839 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:30.920392 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:31.052516 14339 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:31.090818 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:31.261219 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:31.287632 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:31.358871 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:31.415974 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:31.470341 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:31.527837 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:31.550244 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:31.638707 14476 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:31.718492 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:31.835794 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:31.859951 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:31.905530 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:31.934296 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:32.021628 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:32.075659 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:32.078976 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:32.133347 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:10:32.157949 14476 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:32.268204 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:32.274261 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:32.311429 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:32.393334 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:32.501053 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:32.545302 14476 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:32.606339 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:32.732893 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:32.783699 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:32.787854 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:32.834642 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:32.944494 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:32.956606 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:33.056669 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:10:33.122200 14476 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:33.124850 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:33.203554 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:33.221993 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 14:10:33.227953 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:33.260157 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:33.260821 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:33.321354 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:33.327381 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:33.394483 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:33.413465 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 14:10:33.442451 14594 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:33.442575 14594 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:10:33.442960 14594 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:10:33.443321 473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:10:33.443464 473 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:10:33.444016 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:33.444167 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:33.444222 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:10:33.444350 14594 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:33.454905 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:33.493201 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:10:33.493827 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:33.580138 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:33.608198 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:33.667654 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:33.679373 14476 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:33.800161 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:33.903489 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:33.951875 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:34.073298 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:34.130172 14812 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:34.152865 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:34.230490 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:34.250098 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:34.256094 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:34.270709 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:34.319094 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:34.321326 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:34.378724 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:10:34.538815 14476 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:34.612224 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:34.697007 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:34.735936 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:34.822860 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:34.870664 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:34.870764 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:35.012046 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:35.075996 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:35.103909 14476 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:35.172256 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:35.211231 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:35.214604 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:35.269019 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:35.339954 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:35.385140 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:35.406371 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 14:10:35.423957 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:35.476583 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:35.517082 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:35.696942 15405 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:35.698413 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:35.700867 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:35.756237 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:35.823491 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:35.832281 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:35.844210 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:35.855444 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:35.963084 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:35.966993 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:36.113684 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:36.178725 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:36.241214 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:10:36.313616 15546 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:36.352499 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:36.369690 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:36.548861 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:36.643446 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:36.759896 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:36.770771 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:36.789711 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:36.824002 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:36.866946 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:36.875855 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:10:36.917228 15687 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:36.976364 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:37.026719 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:37.244450 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:37.312155 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:37.324010 15546 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:37.324878 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:37.357489 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:37.425426 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:37.442412 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:37.446719 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:37.611696 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:37.694857 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:10:37.714578 15546 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:37.740792 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:37.794637 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:37.808333 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:37.870668 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:37.922370 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:37.944355 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:38.131294 15546 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:38.208320 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:38.250545 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:10:38.251230 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:38.254462 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:38.284013 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:38.285931 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:38.388028 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:38.388566 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:38.467339 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:10:38.468077 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:38.512614 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:38.528856 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 14:10:38.581768 15687 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:38.623448 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:38.633950 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:38.705817 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:10:38.759042 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:38.817095 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:38.967334 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:38.977430 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:39.011713 15687 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:39.055299 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:39.162675 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:39.192808 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:39.360869 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:39.389869 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:39.421967 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:39.464159 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:39.464303 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:39.466625 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:10:39.606130 15687 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:39.722386 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:39.775007 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:39.803246 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:39.807452 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:39.875277 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:40.026779 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:40.125968 15687 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:40.162117 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:40.185963 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:40.254163 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:40.273171 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:40.322444 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:40.356462 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:40.406791 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:40.465226 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:40.487524 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:40.501006 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:40.515264 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:40.522480 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:40.671370 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 22 similar messages]
I20240501 14:10:40.677395 16429 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:40.678205 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:40.753057 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:40.758018 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:40.797187 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:40.797292 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:40.925498 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:41.061245 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:41.093043 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:41.121767 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:41.149168 16429 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:41.158685 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:41.277354 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:41.340229 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:41.504521 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:41.646479 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:41.661172 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:41.699177 16798 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:41.733883 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:41.798606 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:41.835605 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:41.913749 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:41.954062 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:41.980499 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:42.095590 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:42.187834 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:42.264787 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:42.281260 16915 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:42.298863 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:42.329306 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:42.361685 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:42.637943 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:42.652448 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:42.727378 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:42.751318 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:42.762759 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:42.796906 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:42.808754 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:10:42.847429 16798 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:42.867488 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:42.937945 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:42.954803 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:43.078449 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:43.094266 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:43.225875 16915 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:43.226172 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:43.260893 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:43.285605 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:43.311198 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:10:43.382066 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:43.404716 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:43.443679 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:43.500345 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:43.651054 16798 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:43.657337 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:10:43.679870 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:43.721621 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:43.723277 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:43.779781 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:43.824466 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:43.836521 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:43.897887 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:10:43.926317 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:44.033493 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:44.065696 16915 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:44.191489 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:44.209460 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:44.286206 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:44.323527 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:44.347384 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:44.400070 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:44.453925 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:44.505800 16915 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:44.567220 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:44.588884 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:44.705719 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:44.886370 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:44.893553 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:44.911988 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:44.949656 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:44.967988 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:45.021908 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:45.025285 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:45.084395 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:45.113466 16798 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:45.245095 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:45.257360 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:45.278491 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:45.289323 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:45.399477 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:45.408660 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:45.465442 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:45.530786 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:45.658833 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:45.672545 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:10:45.706111 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:45.731159 16798 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:45.766027 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:45.787217 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:45.800545 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:45.894512 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:46.056891 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:46.070219 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:46.109311 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:46.172730 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:46.204893 16798 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:46.251199 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:46.290017 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:46.317152 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:46.377185 15061 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:46.377300 15061 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:10:46.377780 15061 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:10:46.378196 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:10:46.378337 31580 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:10:46.378805 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:46.378911 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:46.378952 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:10:46.379060 15061 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:46.479470 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:46.511626 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:10:46.592429 16798 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:46.614281 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:46.633648 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:46.679491 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:46.785694 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:46.801409 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:46.837205 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:46.921660 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:47.050526 16798 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:47.055563 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:47.164017 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:47.246651 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:47.276072 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:47.319705 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:47.379678 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:10:47.455667 17710 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:47.458029 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:47.459641 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:47.502192 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:47.562944 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:47.635735 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:47.666738 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:47.700088 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:47.761166 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:47.863955 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:47.870877 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:47.872429 16798 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:47.903693 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:48.036257 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:48.048507 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:48.071800 31298 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:48.073524 31429 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:48.074172 31169 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:48.077800 453 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:48.087968 31561 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:48.100587 701 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:48.132360 919 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:09:41.371681 922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:41.371984 922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:41.377700 922 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:09:42.218492 922 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:09:42.652258 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbe6d56762d84933a7b7ddaf8c314b4f: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:42.662375 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 855c959aa35441b989d386caf8f8eded: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:42.662544 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd036c60a12148d2ada73beb29aedcbf: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:42.662659 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 026cd477adec4547bee4fb8004a63348: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:42.662765 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:42.662820 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bd1074f581c447ee86e97b1ef3b8687f: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:42.663165 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1f2efe25014e49d8b36ca467c0f40ef3: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:42.663228 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77c4973914b2412cbc293014310ba9c6: could not find any suitable replica to move from server 6a6fcc099d1340c383e5a429643d5b7c to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:42.663286 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 351ff5b1095247928fbb77893616e4a9: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:42.663345 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01dc7a764a349c9a4a9180993e4fbc3: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:42.663410 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:42.663702 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bd1074f581c447ee86e97b1ef3b8687f: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:42.663763 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd036c60a12148d2ada73beb29aedcbf: could not find any suitable replica to move from server 6a6fcc099d1340c383e5a429643d5b7c to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:42.663807 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1f2efe25014e49d8b36ca467c0f40ef3: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:42.699942 922 rebalancer_tool.cc:1074] tablet 8f91909d3b1b46e9b858ce5c5b2e7975: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '9438789dc61f4501ad1d4ceaeccee458' move scheduled
I20240501 14:09:42.726893 922 rebalancer_tool.cc:1074] tablet 9d2c173b2c32494eb16e46a51ca36e31: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'b934fd09590a40ada094b9715d990a00' move scheduled
I20240501 14:09:42.789887 922 rebalancer_tool.cc:1074] tablet 7a542c07a3f841b0a31051a8a37083bf: '0e4622d80cc94f12b492bdd7597a61e1' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move scheduled
I20240501 14:09:42.875006 922 rebalancer_tool.cc:1074] tablet bca09920a7904362bc546916e3f6f5ac: '6a6fcc099d1340c383e5a429643d5b7c' -> 'a0caf2974041408bb32362bfc4284eaa' move scheduled
I20240501 14:09:43.069242 922 rebalancer_tool.cc:1074] tablet f8cc4d3985224178aa008b4c9de5625b: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '9438789dc61f4501ad1d4ceaeccee458' move scheduled
I20240501 14:09:43.150873 922 rebalancer_tool.cc:1074] tablet b1006490ec2841b2a3365295caaf19e1: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'b934fd09590a40ada094b9715d990a00' move scheduled
I20240501 14:09:43.181387 922 rebalancer_tool.cc:1074] tablet 5aa7bdac7bd1408cbf41d894d40aebd4: '0e4622d80cc94f12b492bdd7597a61e1' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move scheduled
I20240501 14:09:43.265726 922 rebalancer_tool.cc:1074] tablet 0399a419a25c4981a2ee90d76cbf59b1: '6a6fcc099d1340c383e5a429643d5b7c' -> '9438789dc61f4501ad1d4ceaeccee458' move scheduled
I20240501 14:09:43.319604 922 rebalancer_tool.cc:1074] tablet 71241540f585474a8f62da16ed2d9874: 'e7eada5032fb45c3a5e3196e2b3faaab' -> 'a0caf2974041408bb32362bfc4284eaa' move scheduled
I20240501 14:09:43.364246 922 rebalancer_tool.cc:1074] tablet c082ccc777ba4f9a937931f707645e54: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move scheduled
I20240501 14:09:43.423667 922 rebalancer_tool.cc:1074] tablet 72ad9c714b2b4fff94063d2838835173: '0e4622d80cc94f12b492bdd7597a61e1' -> 'b934fd09590a40ada094b9715d990a00' move scheduled
I20240501 14:09:43.511050 922 rebalancer_tool.cc:1074] tablet c0b7ede9ffaf496e9b106b5d05ed25a5: '6a6fcc099d1340c383e5a429643d5b7c' -> '9438789dc61f4501ad1d4ceaeccee458' move scheduled
I20240501 14:09:43.722240 922 rebalancer_tool.cc:1084] tablet 4603d1b1b1d441b1b0be0464e2f2c9e2: 'e7eada5032fb45c3a5e3196e2b3faaab' -> 'a0caf2974041408bb32362bfc4284eaa' move ignored: Illegal state: tablet 4603d1b1b1d441b1b0be0464e2f2c9e2: replica a0caf2974041408bb32362bfc4284eaa is already present
I20240501 14:09:43.722370 922 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:09:45.014253 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd036c60a12148d2ada73beb29aedcbf: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:45.014439 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbe6d56762d84933a7b7ddaf8c314b4f: could not find any suitable replica to move from server 6a6fcc099d1340c383e5a429643d5b7c to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:45.014516 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77c4973914b2412cbc293014310ba9c6: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:45.014829 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd036c60a12148d2ada73beb29aedcbf: could not find any suitable replica to move from server 9438789dc61f4501ad1d4ceaeccee458 to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:45.014907 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:45.014971 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 855c959aa35441b989d386caf8f8eded: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:45.015038 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1f2efe25014e49d8b36ca467c0f40ef3: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:45.015118 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bd1074f581c447ee86e97b1ef3b8687f: could not find any suitable replica to move from server 6a6fcc099d1340c383e5a429643d5b7c to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:45.015182 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77c4973914b2412cbc293014310ba9c6: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:45.319005 922 rebalancer_tool.cc:1084] tablet f26d16499ab54a3e9059f6e4ff041177: '0e4622d80cc94f12b492bdd7597a61e1' -> 'a0caf2974041408bb32362bfc4284eaa' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:09:45.319092 922 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:09:45.917114 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bd1074f581c447ee86e97b1ef3b8687f: could not find any suitable replica to move from server 9438789dc61f4501ad1d4ceaeccee458 to server 0e4622d80cc94f12b492bdd7597a61e1
W20240501 14:09:45.917196 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 026cd477adec4547bee4fb8004a63348: could not find any suitable replica to move from server b934fd09590a40ada094b9715d990a00 to server 30e0b8849e3841ca9ab0de01e719ee4a
W20240501 14:09:45.917227 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01dc7a764a349c9a4a9180993e4fbc3: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:45.917266 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1f2efe25014e49d8b36ca467c0f40ef3: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:45.917330 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server e7eada5032fb45c3a5e3196e2b3faaab
I20240501 14:09:45.986112 922 rebalancer_tool.cc:1074] tablet 5b2e30cb9de14fe19273b3cef801cb67: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '' move scheduled
I20240501 14:09:46.086823 922 rebalancer_tool.cc:1074] tablet 2e68be1521964ba8b864801fdf3b0df3: '6a6fcc099d1340c383e5a429643d5b7c' -> 'a0caf2974041408bb32362bfc4284eaa' move scheduled
I20240501 14:09:46.120106 922 rebalancer_tool.cc:1084] tablet 3c8200d9c20e4140906c71c746badb94: '0e4622d80cc94f12b492bdd7597a61e1' -> 'b934fd09590a40ada094b9715d990a00' move ignored: Illegal state: tablet 3c8200d9c20e4140906c71c746badb94: replica b934fd09590a40ada094b9715d990a00 is already present
I20240501 14:09:46.120194 922 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:09:46.563954 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bd1074f581c447ee86e97b1ef3b8687f: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server 6a6fcc099d1340c383e5a429643d5b7c
I20240501 14:09:46.855115 922 rebalancer_tool.cc:1074] tablet ad0dac76fd0c4ae6b8e0549376c98e5d: '0e4622d80cc94f12b492bdd7597a61e1' -> 'e7eada5032fb45c3a5e3196e2b3faaab' move scheduled
I20240501 14:09:46.907476 922 rebalancer_tool.cc:1074] tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1: '6a6fcc099d1340c383e5a429643d5b7c' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move scheduled
I20240501 14:09:46.941953 922 rebalancer_tool.cc:1084] tablet f26d16499ab54a3e9059f6e4ff041177: '30e0b8849e3841ca9ab0de01e719ee4a' -> '9438789dc61f4501ad1d4ceaeccee458' move ignored: Illegal state: tablet f26d16499ab54a3e9059f6e4ff041177: replica 9438789dc61f4501ad1d4ceaeccee458 is already present
I20240501 14:09:46.942036 922 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:09:47.352545 922 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bd1074f581c447ee86e97b1ef3b8687f: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server 6a6fcc099d1340c383e5a429643d5b7c
I20240501 14:09:47.433697 922 rebalancer_tool.cc:1074] tablet 3c8200d9c20e4140906c71c746badb94: '9438789dc61f4501ad1d4ceaeccee458' -> '0e4622d80cc94f12b492bdd7597a61e1' move scheduled
I20240501 14:09:47.493146 922 rebalancer_tool.cc:1136] tablet 5b2e30cb9de14fe19273b3cef801cb67: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '' move completed: OK
I20240501 14:09:47.565538 922 rebalancer_tool.cc:1136] tablet bca09920a7904362bc546916e3f6f5ac: '6a6fcc099d1340c383e5a429643d5b7c' -> 'a0caf2974041408bb32362bfc4284eaa' move completed: OK
I20240501 14:09:47.577291 922 rebalancer_tool.cc:1136] tablet f8cc4d3985224178aa008b4c9de5625b: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '9438789dc61f4501ad1d4ceaeccee458' move completed: Incomplete: tablet f8cc4d3985224178aa008b4c9de5625b, TS e7eada5032fb45c3a5e3196e2b3faaab -> TS 9438789dc61f4501ad1d4ceaeccee458 move failed, target replica disappeared
I20240501 14:09:47.606842 922 rebalancer_tool.cc:1136] tablet 5aa7bdac7bd1408cbf41d894d40aebd4: '0e4622d80cc94f12b492bdd7597a61e1' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move completed: OK
I20240501 14:09:47.632196 922 rebalancer_tool.cc:1136] tablet 71241540f585474a8f62da16ed2d9874: 'e7eada5032fb45c3a5e3196e2b3faaab' -> 'a0caf2974041408bb32362bfc4284eaa' move completed: OK
I20240501 14:09:47.650177 922 rebalancer_tool.cc:1123] tablet 8f91909d3b1b46e9b858ce5c5b2e7975: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '9438789dc61f4501ad1d4ceaeccee458' move is abandoned: Illegal state: 8f91909d3b1b46e9b858ce5c5b2e7975: source replica e7eada5032fb45c3a5e3196e2b3faaab does not have REPLACE attribute set
I20240501 14:09:47.701082 922 rebalancer_tool.cc:1136] tablet 7a542c07a3f841b0a31051a8a37083bf: '0e4622d80cc94f12b492bdd7597a61e1' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move completed: OK
I20240501 14:09:47.708086 922 rebalancer_tool.cc:1136] tablet 0399a419a25c4981a2ee90d76cbf59b1: '6a6fcc099d1340c383e5a429643d5b7c' -> '9438789dc61f4501ad1d4ceaeccee458' move completed: OK
I20240501 14:09:47.724895 922 rebalancer_tool.cc:1136] tablet b1006490ec2841b2a3365295caaf19e1: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'b934fd09590a40ada094b9715d990a00' move completed: Incomplete: tablet b1006490ec2841b2a3365295caaf19e1, TS 30e0b8849e3841ca9ab0de01e719ee4a -> TS b934fd09590a40ada094b9715d990a00 move failed, target replica disappeared
I20240501 14:09:47.746762 922 rebalancer_tool.cc:1136] tablet 9d2c173b2c32494eb16e46a51ca36e31: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'b934fd09590a40ada094b9715d990a00' move completed: OK
I20240501 14:09:47.760300 922 rebalancer_tool.cc:1136] tablet c082ccc777ba4f9a937931f707645e54: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move completed: OK
I20240501 14:09:47.930163 922 rebalancer_tool.cc:1136] tablet 72ad9c714b2b4fff94063d2838835173: '0e4622d80cc94f12b492bdd7597a61e1' -> 'b934fd09590a40ada094b9715d990a00' move completed: OK
I20240501 14:09:47.977792 922 rebalancer_tool.cc:1074] tablet 4603d1b1b1d441b1b0be0464e2f2c9e2: 'a63e0dd1ba2e4a8db8ace9718a700c32' -> '9438789dc61f4501ad1d4ceaeccee458' move scheduled
I20240501 14:10:47.999459 922 rebalancer_tool.cc:1084] tablet a7810938b8fb406e9c6ac438f54afd03: '9438789dc61f4501ad1d4ceaeccee458' -> 'e7eada5032fb45c3a5e3196e2b3faaab' move ignored: Timed out: BulkChangeConfig RPC to 127.23.191.65:32841 timed out after 60.000s (SENT)
I20240501 14:10:47.999567 922 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): unacceptable health status UNAVAILABLE
W20240501 14:10:48.155421 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:48.184729 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:10:48.288463 17710 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:48.334600 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 14:10:48.335184 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:48.371564 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:48.380712 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:48.428635 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:48.477872 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:48.696527 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 14:10:48.741931 17710 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:48.784816 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:10:48.798914 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:48.806648 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:48.812562 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:48.868327 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:48.959190 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:48.961728 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:49.055541 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:49.056217 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:49.085881 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:49.139871 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:10:49.164000 16798 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:49.245177 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:49.322525 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:49.339130 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:49.386044 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:49.662955 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:49.688660 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:49.724000 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:10:49.732403 18678 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:49.752377 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:49.841835 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:49.901636 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:49.995452 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:50.154568 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:50.168620 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:50.188125 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:50.191084 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:50.237700 16798 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:50.280438 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:50.303344 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:50.309878 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:50.412158 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:50.450882 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:50.489241 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:50.563165 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:50.677417 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:50.684118 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:10:50.699443 18678 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:50.770797 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:50.827204 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:50.854398 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:50.911438 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:51.094755 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 14:10:51.095347 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:51.105295 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:51.192919 18678 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:51.225847 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:51.255720 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:51.387132 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:51.401016 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:51.453316 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:51.473837 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:51.529183 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:51.634938 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:51.680765 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:51.723955 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:51.726490 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:10:51.744553 18678 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:51.817036 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:51.925081 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:51.935320 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:52.009349 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:52.044593 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:52.190863 19214 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:52.190996 19214 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:10:52.191545 19214 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:10:52.191859 472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:10:52.192005 472 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:10:52.193024 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:52.193156 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:10:52.193202 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:10:52.193360 19214 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:52.250044 19270 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:52.255205 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:52.413969 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:52.417335 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:52.504325 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:52.524049 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:52.528896 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:52.588667 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:52.708083 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:52.723786 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:10:52.761108 19390 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:52.798117 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:52.907379 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:52.934924 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:52.935395 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:52.974889 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:53.036329 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:53.128077 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:53.182760 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:53.377056 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:10:53.378166 19390 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:53.388738 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:10:53.389345 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:53.399706 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:53.412515 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:53.478888 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:53.578696 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:53.667901 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:53.737577 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:53.797904 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:53.815512 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:10:53.816185 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:53.818548 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:53.938879 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:53.958391 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:53.962852 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:10:53.964656 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:10:53.975694 19390 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:53.985765 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:54.110881 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:54.168709 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:54.278684 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:54.305949 31034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:10:54.306661 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6f8bd03c05d84fdaaf138a6d0608e30e: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:54.372411 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:54.394973 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:54.437947 19390 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:54.474084 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:54.683071 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:54.749920 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:54.800153 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:54.816787 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:10:54.826779 19390 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:54.877952 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:54.961094 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:55.007700 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:55.080190 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:55.140895 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:55.188407 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:55.206614 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:55.228741 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:55.277993 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:10:55.335574 19538 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:55.422868 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:55.468389 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:55.538530 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:55.546335 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:55.565474 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:55.721207 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:10:55.765501 19390 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:55.916747 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:55.921468 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:55.926817 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:56.022403 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:56.033416 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:56.076085 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:56.087024 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:56.138746 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:56.218838 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:56.291019 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:10:56.294093 20205 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:56.334093 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:56.459725 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 14:10:56.460430 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:56.555131 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:56.579890 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:56.625946 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:56.674391 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:10:56.779081 20205 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:56.828657 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:56.903550 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:56.932440 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:56.990649 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:57.008973 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:57.039862 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:57.158250 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:57.209127 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:57.372850 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:57.375695 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:10:57.375772 20205 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:57.376796 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:57.398293 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:57.557250 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:57.591310 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:57.591609 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:57.595746 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:57.657634 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:57.737752 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:10:57.774044 20431 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:57.913662 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:58.013511 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:58.028985 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:58.037744 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:58.320312 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:58.366837 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:10:58.384850 20431 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:58.439858 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:58.476764 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:58.503958 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 14:10:58.504609 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:58.510689 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:58.574126 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:58.614723 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:58.742664 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:10:58.776013 20431 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:58.837062 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:58.879529 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:58.901757 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:10:58.920922 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:58.947376 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:58.972157 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:10:59.028671 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:59.051977 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:59.088389 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:10:59.223762 20431 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:59.258363 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:59.261637 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:59.421239 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:59.437245 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:59.437835 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:59.464289 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:59.470968 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:59.564083 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:59.613178 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:59.683009 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:10:59.726356 21000 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:10:59.790589 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:59.854759 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:59.854871 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:59.868716 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:10:59.955303 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:10:59.975302 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:00.003645 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:00.047361 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:00.160888 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:00.275971 21139 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:00.308344 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:00.351804 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:00.375377 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:00.553117 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:00.650068 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:00.763931 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:00.798041 21257 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:00.824378 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:00.863080 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:00.893862 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:00.964979 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:01.060885 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:01.248783 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:01.282871 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:01.287212 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:01.292783 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:01.300282 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:01.365875 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:11:01.371425 21257 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:01.394774 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:01.417259 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:01.510560 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:11:01.511211 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:01.597316 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:01.661847 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:01.668824 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:01.710716 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:01.797530 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:01.876638 21513 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:01.965234 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:01.973839 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:01.982342 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:01.984625 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:02.072985 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:02.184072 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:02.200448 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:02.206372 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:02.251148 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:02.303597 21257 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:02.307718 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:02.429049 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:02.466212 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:02.603665 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:02.673465 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:02.693683 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:02.721961 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:11:02.892764 21752 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:02.906417 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:02.920532 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:03.116248 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:03.137306 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:03.221107 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:03.356078 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:03.360653 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:03.378787 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:03.498072 21257 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:03.563897 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:11:03.564534 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:03.626629 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:03.633761 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:03.641189 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:03.662294 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:03.704541 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:03.816807 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:03.845887 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:03.922734 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:03.974157 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:11:04.005695 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:04.007733 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:04.011785 22007 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:04.035998 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:04.069725 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:11:04.070430 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:04.180879 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:04.288129 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:04.393880 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:04.477581 22007 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:04.477660 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:04.509550 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:04.516353 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:04.547914 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:04.553673 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:04.592125 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:04.597402 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:04.678419 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:04.759449 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:04.856989 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:11:04.893033 22007 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:04.896623 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:04.904984 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:05.044438 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:05.079891 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:05.164773 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:05.397791 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:05.431327 22333 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:05.483700 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:05.521797 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:05.534428 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:05.555128 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:05.615870 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:05.704555 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:05.744498 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:05.890646 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:05.897665 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:05.919358 22007 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:05.958880 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:06.033517 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:06.087648 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:06.108234 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:06.154208 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:06.220697 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:11:06.396673 22007 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:06.409665 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:06.417079 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:06.420337 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:06.423139 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:06.520951 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:11:06.601614 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:06.617027 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:06.722245 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:06.768553 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:06.880569 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:11:06.890075 22007 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:06.907729 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:06.957332 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:06.989732 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:06.997184 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:07.135551 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:07.152073 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:07.163194 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:07.173967 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:07.265971 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:11:07.377513 22333 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:07.381063 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:07.474308 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:07.654222 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:07.696796 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:07.738816 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:07.814244 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:07.816128 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:07.968495 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:07.990757 22915 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:08.086820 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:08.118050 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:08.127624 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:08.181908 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:08.229662 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:08.266361 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:08.306871 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:08.393748 22333 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:08.481745 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:08.520963 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:08.559495 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:08.604673 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 14:11:08.605830 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:08.658672 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:08.878886 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:08.886567 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:08.919425 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:08.997002 22915 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:09.050446 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:09.059120 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:09.062045 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:09.124536 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:09.159646 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 14:11:09.195866 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:11:09.201828 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:09.216910 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:09.256181 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:09.369313 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:09.437300 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:09.465595 22915 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:09.470233 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:09.537896 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:09.562974 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:09.569818 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:09.594588 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:09.685353 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:09.715806 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:09.718281 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:09.955929 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:09.964607 23152 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:10.025796 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:10.108001 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:10.116434 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:10.140004 15061 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:10.140130 15061 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:11:10.140704 15061 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:11:10.140973 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:11:10.141116 31580 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:11:10.141904 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:10.142001 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:10.142035 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:11:10.142168 15061 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:10.152981 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:10.221732 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:10.370095 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:10.451653 23152 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:10.575587 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:10.584535 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:10.612224 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:10.722604 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:10.723665 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:10.727515 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:10.893930 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:10.943334 22915 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:10.996862 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:11.044925 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:11.095456 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:11.244865 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:11.246547 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:11.273440 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:11.283077 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:11.283756 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:11.314934 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:11.437386 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:11.458709 23725 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:11.522884 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:11.553386 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:11.590179 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:11.633044 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 14:11:11.633740 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:11.714138 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:11.806506 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:11.820943 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:11.827742 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:11.861721 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:11.864933 23725 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:11.870853 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:11.956132 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:12.104436 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:12.263949 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:12.269601 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:12.293885 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:11:12.347162 23725 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:12.359081 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:12.490234 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:12.521307 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:12.659360 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:12.691468 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:12.780386 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:12.831509 23725 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:12.833773 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:12.903672 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:12.992506 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:13.131996 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:13.150599 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:13.222153 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:13.257429 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:13.313324 23725 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:13.357578 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:13.511965 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:13.512954 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:13.548712 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:13.587841 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:13.622730 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 14:11:13.731778 22915 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:13.761197 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:13.806397 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:13.812757 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:13.894426 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:13.913811 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:13.985836 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:13.993130 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:14.009645 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:14.074301 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:14.117650 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:14.144753 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:14.246157 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:14.246585 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 14:11:14.247220 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:14.293535 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 14:11:14.299392 24392 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:14.351454 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:14.475697 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:14.492179 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:14.507226 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:14.524730 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:14.568589 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:14.636152 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:14.638012 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:14.727496 24392 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:14.804401 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:14.874884 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:14.883201 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:14.905325 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:14.928014 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:15.068064 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:15.119046 24392 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:15.140022 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:15.170799 15061 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:15.170938 15061 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:11:15.171422 15061 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:11:15.172591 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:11:15.172762 31580 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:11:15.173594 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:15.173707 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:15.173744 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:11:15.174346 15061 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:15.257350 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:15.341248 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:15.465207 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:15.467391 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:15.545877 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:15.593389 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:15.637795 24697 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:15.654608 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:15.787964 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:15.957279 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:15.972421 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:15.988498 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:16.059788 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:16.150934 24697 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:16.181092 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:16.239266 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:16.256419 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:16.389883 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:16.413151 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:16.414429 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:16.430984 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:16.431980 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:16.470124 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:16.471299 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:16.472963 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:16.590906 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:16.631911 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:16.646415 24815 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:16.812299 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 14:11:16.812319 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:16.817819 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:16.841807 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:16.937878 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:16.994688 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:17.102592 24815 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:17.152707 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:17.200297 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:17.210492 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:17.226884 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:17.237346 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:17.291024 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:17.331568 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:17.416579 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:17.425030 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:17.546066 24697 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:17.631506 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:17.680282 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:17.723035 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:17.805882 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:17.959990 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:18.084093 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:18.114430 24815 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:18.139833 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:18.267045 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:18.274103 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:18.363121 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:18.423172 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:18.575201 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:18.625655 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 14:11:18.668185 24697 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:18.708372 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:18.725703 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:18.759706 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:18.761161 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:18.823653 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:18.834846 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:18.896906 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:18.973623 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:19.015368 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:19.029616 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:19.043931 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:19.059149 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:19.105751 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:11:19.129989 24697 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:19.249140 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:11:19.295575 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:19.331192 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:19.332003 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:19.373346 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:19.512455 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:19.579119 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:11:19.609830 24697 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:19.615361 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:19.617921 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:19.649222 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:19.680879 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:19.759555 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:19.819669 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:19.875625 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:19.885430 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:19.935330 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:19.971136 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:20.002164 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:20.221855 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:20.234531 24697 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:20.394444 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:20.407908 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:20.491358 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:20.660969 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:20.705332 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:20.746210 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:20.772042 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:11:20.802357 24697 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:20.879889 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:20.953935 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:21.081213 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:21.239372 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:21.259495 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:21.261884 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:21.272763 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:21.273998 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:21.306977 24697 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:21.474612 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:21.476610 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:21.489058 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:21.489137 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:21.508961 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:21.528053 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:21.680672 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:21.745146 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:21.834836 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:11:21.835638 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:11:21.840553 24697 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:21.983299 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:21.991619 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:22.025190 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:22.088444 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:22.093691 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:22.105844 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:22.107450 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:22.162657 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:22.296609 25884 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:22.425977 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:22.482893 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:22.504920 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:22.530042 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:22.593832 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:22.606647 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:22.661487 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:22.790702 25884 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:22.947475 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:23.025985 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:23.028259 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:23.097393 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:23.135964 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:11:23.278730 24697 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:23.340281 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:23.404210 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:23.430970 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:23.520748 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:23.531395 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:23.551347 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:23.612351 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:23.649083 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:23.703382 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:23.822793 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:11:23.823436 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:23.832602 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:23.860770 26605 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:23.880738 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:23.958052 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:23.990757 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:24.054668 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:24.063583 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:24.148808 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:24.162438 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:24.256301 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:24.288468 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:11:24.297953 26606 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:24.301380 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:11:24.361267 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:24.383630 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:24.419670 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:24.457039 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:24.532369 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:24.618512 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:24.622655 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:24.714319 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:11:24.749702 26606 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:24.771183 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:11:24.806689 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:24.856577 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:24.954605 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:24.956874 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:24.995612 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:25.003144 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:25.086122 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:25.311339 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:25.322780 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:25.372749 26941 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:25.448508 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:25.593173 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:25.661927 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:25.777729 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:25.798415 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:25.845523 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:25.924543 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:25.931480 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:25.971758 27087 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:26.130615 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:26.132812 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:26.197683 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:26.250303 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:26.303510 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:26.326711 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:26.393476 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:26.413658 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:26.505879 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:26.524287 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:11:26.544636 27223 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:26.553397 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:26.661870 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:26.709225 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:26.743362 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:26.857200 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:26.862133 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:11:26.863094 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:26.876053 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:26.917330 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:26.971067 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:27.011842 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:27.165637 27087 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:27.195899 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:27.227555 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:27.315308 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:27.369390 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:27.397226 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:27.446337 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:27.500767 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:27.507007 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:27.657115 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:11:27.730752 27087 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:27.754202 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:27.886871 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:27.927227 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:27.947065 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:28.061612 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:11:28.128877 27492 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:28.329574 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:28.336138 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:28.400180 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:28.453631 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:28.486891 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:28.504880 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:11:28.559353 27492 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:28.611352 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:28.721637 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:28.725946 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:28.791246 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:28.897244 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 14:11:28.897881 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:11:28.935835 27087 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:28.950294 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:28.956357 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:28.968984 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:29.006068 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:29.007568 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:29.182696 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:29.195333 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:29.254873 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:29.290927 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:29.331037 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:29.331298 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:11:29.331938 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:29.364874 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:29.426602 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:11:29.427222 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:29.429075 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:11:29.443281 27890 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:29.472445 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:29.729930 31036 maintenance_manager.cc:419] P d913062e66274883a568f59820e333ee: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033368
I20240501 14:11:29.752169 31035 maintenance_manager.cc:643] P d913062e66274883a568f59820e333ee: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.022s user 0.020s sys 0.000s Metrics: {"bytes_written":20974,"cfile_init":1,"dirs.queue_time_us":154,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1333,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":728,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":43,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":163}
I20240501 14:11:29.753022 31036 maintenance_manager.cc:419] P d913062e66274883a568f59820e333ee: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 11878 bytes on disk
I20240501 14:11:29.753502 31035 maintenance_manager.cc:643] P d913062e66274883a568f59820e333ee: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
W20240501 14:11:29.785523 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:29.786092 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:29.799422 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:29.871254 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:29.920578 27087 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:29.991885 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:29.998953 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:30.026077 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:30.089704 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:30.195405 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:30.292604 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:30.325464 27087 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:30.425616 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:30.591223 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:30.595185 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:30.717600 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:30.744199 27087 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:30.796033 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:30.974572 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:31.009732 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:31.039831 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:31.042074 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:31.083441 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:31.114262 27239 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:31.114392 27239 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:11:31.114936 27239 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:11:31.115259 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:11:31.115417 31581 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:11:31.116094 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:31.116190 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:31.116235 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:11:31.116360 27239 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:31.119788 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:31.148211 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:11:31.233186 27087 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:31.259460 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:31.276048 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:31.417721 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:31.507001 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:31.514940 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:31.608654 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:31.616488 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:31.624634 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:31.626025 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:31.663182 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:31.697371 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:31.719321 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:31.761298 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:31.794806 27087 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:31.811816 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:31.960615 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 14:11:32.097340 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:32.117722 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:32.166541 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:32.207753 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:32.209224 27087 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:32.220875 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:32.288883 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:32.289995 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:32.448982 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:32.452497 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:32.562755 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:32.565631 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:32.637184 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:32.672446 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:32.801918 27087 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:32.976754 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:33.060264 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:33.111089 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:33.253255 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:33.260202 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:33.333619 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:33.342936 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:33.353374 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:11:33.355908 27087 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:33.461827 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:33.506989 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:33.622702 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:33.643256 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:33.706207 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:33.732695 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:33.783210 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:33.875339 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:33.877511 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:33.966917 28936 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:34.038348 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:11:34.077075 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:34.102622 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:34.168242 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:34.211591 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:34.229830 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:34.242616 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:34.291340 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:34.375799 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:34.403929 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:34.411365 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:34.431335 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 14:11:34.468765 28936 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:34.494741 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:11:34.612200 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:34.633608 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:34.656565 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:34.684377 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:34.724715 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:34.793423 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:34.797773 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:34.839435 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:34.840360 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:11:34.917292 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(5b2e30cb9de14fe19273b3cef801cb67): perf score=0.033344
I20240501 14:11:34.938886 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling FlushMRSOp(f26d16499ab54a3e9059f6e4ff041177): perf score=0.033348
I20240501 14:11:34.951634 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(5b2e30cb9de14fe19273b3cef801cb67) complete. Timing: real 0.034s user 0.024s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":179,"dirs.queue_time_us":260,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":966,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":448,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":217,"threads_started":3,"wal-append.queue_time_us":150}
I20240501 14:11:34.952942 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(5b2e30cb9de14fe19273b3cef801cb67): 4834 bytes on disk
I20240501 14:11:34.953505 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(5b2e30cb9de14fe19273b3cef801cb67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240501 14:11:34.964934 28936 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:11:34.967833 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: FlushMRSOp(f26d16499ab54a3e9059f6e4ff041177) complete. Timing: real 0.029s user 0.021s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":208,"dirs.queue_time_us":288,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1133,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":402,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":369,"threads_started":3,"wal-append.queue_time_us":161}
I20240501 14:11:34.969205 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling UndoDeltaBlockGCOp(f26d16499ab54a3e9059f6e4ff041177): 4579 bytes on disk
I20240501 14:11:34.969720 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: UndoDeltaBlockGCOp(f26d16499ab54a3e9059f6e4ff041177) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240501 14:11:35.018733 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling FlushMRSOp(7a542c07a3f841b0a31051a8a37083bf): perf score=0.033372
W20240501 14:11:35.070261 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:11:35.087531 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling FlushMRSOp(f26d16499ab54a3e9059f6e4ff041177): perf score=0.033392
I20240501 14:11:35.098137 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: FlushMRSOp(7a542c07a3f841b0a31051a8a37083bf) complete. Timing: real 0.079s user 0.033s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":8565,"dirs.queue_time_us":14709,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1109,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":631,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":270,"threads_started":3,"wal-append.queue_time_us":780}
I20240501 14:11:35.099506 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(7a542c07a3f841b0a31051a8a37083bf): 4921 bytes on disk
I20240501 14:11:35.100193 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(7a542c07a3f841b0a31051a8a37083bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240501 14:11:35.126487 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: FlushMRSOp(f26d16499ab54a3e9059f6e4ff041177) complete. Timing: real 0.038s user 0.030s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":25130,"dirs.queue_time_us":6907,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":575,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":256,"threads_started":3,"wal-append.queue_time_us":369}
I20240501 14:11:35.128125 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling UndoDeltaBlockGCOp(f26d16499ab54a3e9059f6e4ff041177): 4579 bytes on disk
I20240501 14:11:35.128746 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: UndoDeltaBlockGCOp(f26d16499ab54a3e9059f6e4ff041177) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 14:11:35.129597 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling FlushMRSOp(7a542c07a3f841b0a31051a8a37083bf): perf score=0.033402
I20240501 14:11:35.173808 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: FlushMRSOp(7a542c07a3f841b0a31051a8a37083bf) complete. Timing: real 0.044s user 0.020s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":7665,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1472,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":608,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":123,"threads_started":1,"wal-append.queue_time_us":9757}
I20240501 14:11:35.174914 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling UndoDeltaBlockGCOp(7a542c07a3f841b0a31051a8a37083bf): 4921 bytes on disk
I20240501 14:11:35.175637 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: UndoDeltaBlockGCOp(7a542c07a3f841b0a31051a8a37083bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240501 14:11:35.177574 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:35.177757 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:35.185382 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:35.295234 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:35.432426 29043 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:35.564661 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:35.602583 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling FlushMRSOp(ea66fe3d0da04d18b961927a9c988a20): perf score=0.033364
I20240501 14:11:35.614439 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: FlushMRSOp(ea66fe3d0da04d18b961927a9c988a20) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":195,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1512,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":469,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":161}
I20240501 14:11:35.615361 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(ea66fe3d0da04d18b961927a9c988a20): 4581 bytes on disk
I20240501 14:11:35.615778 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(ea66fe3d0da04d18b961927a9c988a20) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240501 14:11:35.625589 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:35.677717 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling FlushMRSOp(0399a419a25c4981a2ee90d76cbf59b1): perf score=0.033384
I20240501 14:11:35.689745 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: FlushMRSOp(0399a419a25c4981a2ee90d76cbf59b1) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":169,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":1461,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":438,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":134}
I20240501 14:11:35.690656 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling UndoDeltaBlockGCOp(0399a419a25c4981a2ee90d76cbf59b1): 4834 bytes on disk
I20240501 14:11:35.691155 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: UndoDeltaBlockGCOp(0399a419a25c4981a2ee90d76cbf59b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240501 14:11:35.715592 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:35.722327 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling FlushMRSOp(9d2c173b2c32494eb16e46a51ca36e31): perf score=0.033396
I20240501 14:11:35.734474 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: FlushMRSOp(9d2c173b2c32494eb16e46a51ca36e31) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":161,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":1387,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":412,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":161,"threads_started":2,"wal-append.queue_time_us":180}
I20240501 14:11:35.735193 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling UndoDeltaBlockGCOp(9d2c173b2c32494eb16e46a51ca36e31): 4921 bytes on disk
I20240501 14:11:35.735529 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: UndoDeltaBlockGCOp(9d2c173b2c32494eb16e46a51ca36e31) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240501 14:11:35.785424 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:35.818459 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:35.857499 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:11:35.935817 29382 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:35.942162 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:36.007611 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:36.030092 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:36.081185 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:11:36.117820 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling FlushMRSOp(ad0dac76fd0c4ae6b8e0549376c98e5d): perf score=0.033375
W20240501 14:11:36.118124 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:36.130203 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: FlushMRSOp(ad0dac76fd0c4ae6b8e0549376c98e5d) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":162,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1398,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":132}
I20240501 14:11:36.130977 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(ad0dac76fd0c4ae6b8e0549376c98e5d): 4581 bytes on disk
I20240501 14:11:36.131335 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(ad0dac76fd0c4ae6b8e0549376c98e5d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240501 14:11:36.193289 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling FlushMRSOp(3c8200d9c20e4140906c71c746badb94): perf score=0.033396
I20240501 14:11:36.208498 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: FlushMRSOp(3c8200d9c20e4140906c71c746badb94) complete. Timing: real 0.015s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":141,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":1702,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":535,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":127,"threads_started":2,"wal-append.queue_time_us":143}
I20240501 14:11:36.209493 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling FlushMRSOp(8f91909d3b1b46e9b858ce5c5b2e7975): perf score=0.033399
W20240501 14:11:36.215679 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:11:36.224565 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: FlushMRSOp(8f91909d3b1b46e9b858ce5c5b2e7975) complete. Timing: real 0.015s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1054,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1423,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":464,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":47,"threads_started":1,"wal-append.queue_time_us":116}
I20240501 14:11:36.225415 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling UndoDeltaBlockGCOp(3c8200d9c20e4140906c71c746badb94): 4921 bytes on disk
I20240501 14:11:36.225781 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: UndoDeltaBlockGCOp(3c8200d9c20e4140906c71c746badb94) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 14:11:36.226404 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling UndoDeltaBlockGCOp(8f91909d3b1b46e9b858ce5c5b2e7975): 4834 bytes on disk
I20240501 14:11:36.226979 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: UndoDeltaBlockGCOp(8f91909d3b1b46e9b858ce5c5b2e7975) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240501 14:11:36.238024 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:36.251745 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:11:36.326562 29382 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:36.328377 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:36.413255 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:36.444576 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:36.508309 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:36.513474 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:36.583374 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:11:36.633440 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling FlushMRSOp(2e68be1521964ba8b864801fdf3b0df3): perf score=0.033339
I20240501 14:11:36.648931 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: FlushMRSOp(2e68be1521964ba8b864801fdf3b0df3) complete. Timing: real 0.015s user 0.013s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":315,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1549,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":495,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":176,"threads_started":2,"wal-append.queue_time_us":133}
I20240501 14:11:36.649847 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(2e68be1521964ba8b864801fdf3b0df3): 4834 bytes on disk
I20240501 14:11:36.650354 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(2e68be1521964ba8b864801fdf3b0df3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240501 14:11:36.651005 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling FlushMRSOp(8ffdb76b2c5149a18d45bee9581e605a): perf score=0.033343
W20240501 14:11:36.662351 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:11:36.662789 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: FlushMRSOp(8ffdb76b2c5149a18d45bee9581e605a) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":1311,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":382,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":225}
I20240501 14:11:36.663719 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(8ffdb76b2c5149a18d45bee9581e605a): 4581 bytes on disk
I20240501 14:11:36.664075 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(8ffdb76b2c5149a18d45bee9581e605a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240501 14:11:36.672354 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:36.680601 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:36.698547 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:11:36.710073 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(8ffdb76b2c5149a18d45bee9581e605a): perf score=0.033362
I20240501 14:11:36.722360 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(8ffdb76b2c5149a18d45bee9581e605a) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":183,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1444,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":459,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":154}
I20240501 14:11:36.723172 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(c0b7ede9ffaf496e9b106b5d05ed25a5): perf score=0.033364
I20240501 14:11:36.729105 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling FlushMRSOp(c0b7ede9ffaf496e9b106b5d05ed25a5): perf score=0.033367
I20240501 14:11:36.736506 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(c0b7ede9ffaf496e9b106b5d05ed25a5) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":1175,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":482,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":96,"threads_started":1,"wal-append.queue_time_us":166}
I20240501 14:11:36.737540 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(c0b7ede9ffaf496e9b106b5d05ed25a5): 4921 bytes on disk
I20240501 14:11:36.737895 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(c0b7ede9ffaf496e9b106b5d05ed25a5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240501 14:11:36.738723 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(8ffdb76b2c5149a18d45bee9581e605a): 4581 bytes on disk
I20240501 14:11:36.738956 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling FlushMRSOp(2e68be1521964ba8b864801fdf3b0df3): perf score=0.033371
I20240501 14:11:36.739321 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(8ffdb76b2c5149a18d45bee9581e605a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 14:11:36.750231 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: FlushMRSOp(c0b7ede9ffaf496e9b106b5d05ed25a5) complete. Timing: real 0.021s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":234,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1152,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":447,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":124,"threads_started":2,"wal-append.queue_time_us":3786}
I20240501 14:11:36.751051 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling UndoDeltaBlockGCOp(c0b7ede9ffaf496e9b106b5d05ed25a5): 4921 bytes on disk
I20240501 14:11:36.751477 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: UndoDeltaBlockGCOp(c0b7ede9ffaf496e9b106b5d05ed25a5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 14:11:36.753016 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: FlushMRSOp(2e68be1521964ba8b864801fdf3b0df3) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":162,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1154,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":525,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":138,"threads_started":2,"wal-append.queue_time_us":125}
I20240501 14:11:36.754007 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling UndoDeltaBlockGCOp(2e68be1521964ba8b864801fdf3b0df3): 4834 bytes on disk
I20240501 14:11:36.754395 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: UndoDeltaBlockGCOp(2e68be1521964ba8b864801fdf3b0df3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240501 14:11:36.771797 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:36.843290 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:36.844787 29600 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:36.864761 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:36.934783 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:36.990092 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:37.059384 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:37.099682 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:37.203410 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:11:37.225697 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:11:37.241472 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(28013d7df97943e09c27c3bb1794ee80): perf score=0.033354
I20240501 14:11:37.253863 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(28013d7df97943e09c27c3bb1794ee80) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":152,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1410,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":469,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":162,"threads_started":2,"wal-append.queue_time_us":159}
I20240501 14:11:37.254716 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(28013d7df97943e09c27c3bb1794ee80): 4919 bytes on disk
I20240501 14:11:37.255059 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(28013d7df97943e09c27c3bb1794ee80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240501 14:11:37.256124 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling FlushMRSOp(72ad9c714b2b4fff94063d2838835173): perf score=0.033356
I20240501 14:11:37.267781 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: FlushMRSOp(72ad9c714b2b4fff94063d2838835173) complete. Timing: real 0.011s user 0.007s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":197,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1230,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":483,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":175,"threads_started":2,"wal-append.queue_time_us":139}
I20240501 14:11:37.268745 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling FlushMRSOp(28013d7df97943e09c27c3bb1794ee80): perf score=0.033358
I20240501 14:11:37.280464 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: FlushMRSOp(28013d7df97943e09c27c3bb1794ee80) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":1409,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":382,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":66,"threads_started":1,"wal-append.queue_time_us":147}
I20240501 14:11:37.281271 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling UndoDeltaBlockGCOp(28013d7df97943e09c27c3bb1794ee80): 4919 bytes on disk
I20240501 14:11:37.281663 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: UndoDeltaBlockGCOp(28013d7df97943e09c27c3bb1794ee80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 14:11:37.282263 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling UndoDeltaBlockGCOp(72ad9c714b2b4fff94063d2838835173): 4834 bytes on disk
I20240501 14:11:37.282552 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: UndoDeltaBlockGCOp(72ad9c714b2b4fff94063d2838835173) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 14:11:37.352303 29600 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:37.356235 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:37.358374 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:11:37.416668 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling FlushMRSOp(5aa7bdac7bd1408cbf41d894d40aebd4): perf score=0.033403
I20240501 14:11:37.428194 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: FlushMRSOp(5aa7bdac7bd1408cbf41d894d40aebd4) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":177,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":1301,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":454,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":125}
I20240501 14:11:37.428972 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(5aa7bdac7bd1408cbf41d894d40aebd4): 4581 bytes on disk
I20240501 14:11:37.429335 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(5aa7bdac7bd1408cbf41d894d40aebd4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240501 14:11:37.482913 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:37.558990 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:37.643287 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:37.680544 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling FlushMRSOp(f8cc4d3985224178aa008b4c9de5625b): perf score=0.033343
I20240501 14:11:37.692226 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: FlushMRSOp(f8cc4d3985224178aa008b4c9de5625b) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1621,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":452,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":88,"threads_started":1,"wal-append.queue_time_us":187}
I20240501 14:11:37.693147 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling FlushMRSOp(6bcd6a2f3c014d5ca0e32e16cdbc4edc): perf score=0.033345
I20240501 14:11:37.706193 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: FlushMRSOp(6bcd6a2f3c014d5ca0e32e16cdbc4edc) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1301,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":389,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":70,"threads_started":1,"wal-append.queue_time_us":216}
I20240501 14:11:37.706913 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(f8cc4d3985224178aa008b4c9de5625b): 4921 bytes on disk
I20240501 14:11:37.707255 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(f8cc4d3985224178aa008b4c9de5625b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240501 14:11:37.707877 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(6bcd6a2f3c014d5ca0e32e16cdbc4edc): 4581 bytes on disk
I20240501 14:11:37.708178 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(6bcd6a2f3c014d5ca0e32e16cdbc4edc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 14:11:37.754791 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling FlushMRSOp(f8cc4d3985224178aa008b4c9de5625b): perf score=0.033367
I20240501 14:11:37.757428 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(a7810938b8fb406e9c6ac438f54afd03): perf score=0.033364
I20240501 14:11:37.767316 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: FlushMRSOp(f8cc4d3985224178aa008b4c9de5625b) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":160,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1727,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":425,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":129,"threads_started":2,"wal-append.queue_time_us":128}
I20240501 14:11:37.768105 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling UndoDeltaBlockGCOp(f8cc4d3985224178aa008b4c9de5625b): 4921 bytes on disk
I20240501 14:11:37.768458 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: UndoDeltaBlockGCOp(f8cc4d3985224178aa008b4c9de5625b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240501 14:11:37.769124 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(a7810938b8fb406e9c6ac438f54afd03) complete. Timing: real 0.011s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1181,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":443,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":896,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":125}
I20240501 14:11:37.769994 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(a7810938b8fb406e9c6ac438f54afd03): 4834 bytes on disk
I20240501 14:11:37.770383 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(a7810938b8fb406e9c6ac438f54afd03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240501 14:11:37.784504 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling FlushMRSOp(6bcd6a2f3c014d5ca0e32e16cdbc4edc): perf score=0.033369
I20240501 14:11:37.797375 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: FlushMRSOp(6bcd6a2f3c014d5ca0e32e16cdbc4edc) complete. Timing: real 0.013s user 0.011s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":155,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1332,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":453,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":139}
I20240501 14:11:37.798163 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling UndoDeltaBlockGCOp(6bcd6a2f3c014d5ca0e32e16cdbc4edc): 4581 bytes on disk
I20240501 14:11:37.798475 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: UndoDeltaBlockGCOp(6bcd6a2f3c014d5ca0e32e16cdbc4edc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240501 14:11:37.844744 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:11:37.917699 29869 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:37.968317 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:38.039153 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:38.095979 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:38.169010 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:38.270601 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling FlushMRSOp(b67bd0eb45a3486492b75ca3ad3be524): perf score=0.033348
I20240501 14:11:38.272809 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(bca09920a7904362bc546916e3f6f5ac): perf score=0.033351
I20240501 14:11:38.288079 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: FlushMRSOp(b67bd0eb45a3486492b75ca3ad3be524) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2160,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1364,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":532,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":143,"threads_started":2,"wal-append.queue_time_us":146}
I20240501 14:11:38.289067 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling UndoDeltaBlockGCOp(b67bd0eb45a3486492b75ca3ad3be524): 4834 bytes on disk
I20240501 14:11:38.289534 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: UndoDeltaBlockGCOp(b67bd0eb45a3486492b75ca3ad3be524) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 14:11:38.291208 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(bca09920a7904362bc546916e3f6f5ac) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":1202,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":524,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":148,"threads_started":2,"wal-append.queue_time_us":594}
I20240501 14:11:38.292270 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(96f5e4e25a674a99a6762757f0ab1545): perf score=0.033356
I20240501 14:11:38.300380 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling FlushMRSOp(96f5e4e25a674a99a6762757f0ab1545): perf score=0.033356
I20240501 14:11:38.305642 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(96f5e4e25a674a99a6762757f0ab1545) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1308,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":482,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":83,"threads_started":1,"wal-append.queue_time_us":153}
I20240501 14:11:38.306659 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(bca09920a7904362bc546916e3f6f5ac): 4921 bytes on disk
I20240501 14:11:38.307031 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(bca09920a7904362bc546916e3f6f5ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 14:11:38.307929 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(96f5e4e25a674a99a6762757f0ab1545): 4581 bytes on disk
I20240501 14:11:38.308343 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(96f5e4e25a674a99a6762757f0ab1545) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 14:11:38.312171 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: FlushMRSOp(96f5e4e25a674a99a6762757f0ab1545) complete. Timing: real 0.012s user 0.006s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":1203,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":413,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":143,"threads_started":2,"wal-append.queue_time_us":160}
I20240501 14:11:38.312992 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling UndoDeltaBlockGCOp(96f5e4e25a674a99a6762757f0ab1545): 4581 bytes on disk
I20240501 14:11:38.313318 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: UndoDeltaBlockGCOp(96f5e4e25a674a99a6762757f0ab1545) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240501 14:11:38.359302 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:38.423615 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:11:38.460786 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling FlushMRSOp(b67bd0eb45a3486492b75ca3ad3be524): perf score=0.033399
I20240501 14:11:38.475385 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: FlushMRSOp(b67bd0eb45a3486492b75ca3ad3be524) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":147,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1334,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":500,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":358}
I20240501 14:11:38.476135 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(b67bd0eb45a3486492b75ca3ad3be524): 4834 bytes on disk
I20240501 14:11:38.476469 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(b67bd0eb45a3486492b75ca3ad3be524) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240501 14:11:38.509788 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:11:38.530848 29600 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:38.546587 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:38.564525 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:38.572760 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:38.634780 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:38.659680 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:11:38.791805 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling FlushMRSOp(71241540f585474a8f62da16ed2d9874): perf score=0.033337
I20240501 14:11:38.804155 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: FlushMRSOp(71241540f585474a8f62da16ed2d9874) complete. Timing: real 0.012s user 0.009s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":165,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1393,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":490,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":138,"threads_started":2,"wal-append.queue_time_us":130}
I20240501 14:11:38.804967 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling UndoDeltaBlockGCOp(71241540f585474a8f62da16ed2d9874): 4921 bytes on disk
I20240501 14:11:38.805322 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: UndoDeltaBlockGCOp(71241540f585474a8f62da16ed2d9874) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240501 14:11:38.810496 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(71241540f585474a8f62da16ed2d9874): perf score=0.033341
I20240501 14:11:38.815351 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling FlushMRSOp(c082ccc777ba4f9a937931f707645e54): perf score=0.033338
W20240501 14:11:38.822870 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:38.825604 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(71241540f585474a8f62da16ed2d9874) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":186,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":1684,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":519,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":161}
I20240501 14:11:38.827505 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(043344ce36a74a18a0d67ff5bd473e68): perf score=0.033343
I20240501 14:11:38.834833 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: FlushMRSOp(c082ccc777ba4f9a937931f707645e54) complete. Timing: real 0.019s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":239,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1354,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":632,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":153}
I20240501 14:11:38.835863 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling UndoDeltaBlockGCOp(c082ccc777ba4f9a937931f707645e54): 4834 bytes on disk
I20240501 14:11:38.836269 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: UndoDeltaBlockGCOp(c082ccc777ba4f9a937931f707645e54) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240501 14:11:38.842016 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(043344ce36a74a18a0d67ff5bd473e68) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":1329,"drs_written":1,"lbm_read_time_us":32,"lbm_reads_lt_1ms":4,"lbm_write_time_us":463,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":70,"threads_started":1,"wal-append.queue_time_us":149}
I20240501 14:11:38.842921 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(71241540f585474a8f62da16ed2d9874): 4921 bytes on disk
I20240501 14:11:38.843247 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(71241540f585474a8f62da16ed2d9874) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240501 14:11:38.843854 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(043344ce36a74a18a0d67ff5bd473e68): 4581 bytes on disk
I20240501 14:11:38.844158 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(043344ce36a74a18a0d67ff5bd473e68) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240501 14:11:38.867008 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:38.886237 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:11:38.978679 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling FlushMRSOp(043344ce36a74a18a0d67ff5bd473e68): perf score=0.033386
I20240501 14:11:38.990473 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: FlushMRSOp(043344ce36a74a18a0d67ff5bd473e68) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":163,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1516,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":417,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":139}
I20240501 14:11:38.991329 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(043344ce36a74a18a0d67ff5bd473e68): 4581 bytes on disk
I20240501 14:11:38.991752 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(043344ce36a74a18a0d67ff5bd473e68) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240501 14:11:38.993613 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:39.005496 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:39.018378 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:39.096307 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:11:39.096967 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:11:39.109856 29600 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:39.116591 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:39.116703 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:39.181805 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:39.252979 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:39.262120 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:11:39.307232 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling FlushMRSOp(5d994081c3414f779ba2cae431cb1d97): perf score=0.033378
I20240501 14:11:39.319779 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: FlushMRSOp(5d994081c3414f779ba2cae431cb1d97) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1498,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":457,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":140}
I20240501 14:11:39.320554 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling FlushMRSOp(6f8bd03c05d84fdaaf138a6d0608e30e): perf score=0.033381
I20240501 14:11:39.331552 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: FlushMRSOp(6f8bd03c05d84fdaaf138a6d0608e30e) complete. Timing: real 0.011s user 0.009s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1407,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":381,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":192}
I20240501 14:11:39.332322 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling UndoDeltaBlockGCOp(5d994081c3414f779ba2cae431cb1d97): 4921 bytes on disk
I20240501 14:11:39.332679 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: UndoDeltaBlockGCOp(5d994081c3414f779ba2cae431cb1d97) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240501 14:11:39.333277 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling UndoDeltaBlockGCOp(6f8bd03c05d84fdaaf138a6d0608e30e): 4581 bytes on disk
I20240501 14:11:39.333596 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: UndoDeltaBlockGCOp(6f8bd03c05d84fdaaf138a6d0608e30e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 14:11:39.338263 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling FlushMRSOp(d265eff200b645bfac8f8906654385f6): perf score=0.033383
I20240501 14:11:39.346380 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(d265eff200b645bfac8f8906654385f6): perf score=0.033388
I20240501 14:11:39.351575 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: FlushMRSOp(d265eff200b645bfac8f8906654385f6) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":222,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1380,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":161,"threads_started":2,"wal-append.queue_time_us":168}
I20240501 14:11:39.352421 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling UndoDeltaBlockGCOp(d265eff200b645bfac8f8906654385f6): 4834 bytes on disk
I20240501 14:11:39.352823 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: UndoDeltaBlockGCOp(d265eff200b645bfac8f8906654385f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 14:11:39.361351 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(d265eff200b645bfac8f8906654385f6) complete. Timing: real 0.015s user 0.013s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":187,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1257,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":500,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":174,"threads_started":2,"wal-append.queue_time_us":259}
I20240501 14:11:39.362326 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(d265eff200b645bfac8f8906654385f6): 4834 bytes on disk
I20240501 14:11:39.362735 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(d265eff200b645bfac8f8906654385f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 14:11:39.363502 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(6f8bd03c05d84fdaaf138a6d0608e30e): perf score=0.033391
W20240501 14:11:39.368484 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:39.378360 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:39.382537 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:11:39.383239 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(6f8bd03c05d84fdaaf138a6d0608e30e) complete. Timing: real 0.020s user 0.013s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1361,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":505,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":93,"threads_started":1,"wal-append.queue_time_us":2672}
I20240501 14:11:39.384268 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(6f8bd03c05d84fdaaf138a6d0608e30e): 4581 bytes on disk
I20240501 14:11:39.384666 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(6f8bd03c05d84fdaaf138a6d0608e30e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240501 14:11:39.419878 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:39.468483 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:39.531690 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:11:39.533793 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:11:39.545347 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:39.589663 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:11:39.637183 29600 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:39.680480 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:39.728030 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:39.748891 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:39.809530 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:39.911182 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:40.010476 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:11:40.086231 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling FlushMRSOp(4603d1b1b1d441b1b0be0464e2f2c9e2): perf score=0.033347
I20240501 14:11:40.094763 30298 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:11:40.098205 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: FlushMRSOp(4603d1b1b1d441b1b0be0464e2f2c9e2) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":258,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1438,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":479,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":159,"threads_started":2,"wal-append.queue_time_us":170}
I20240501 14:11:40.098961 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling UndoDeltaBlockGCOp(4603d1b1b1d441b1b0be0464e2f2c9e2): 4581 bytes on disk
I20240501 14:11:40.099339 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: UndoDeltaBlockGCOp(4603d1b1b1d441b1b0be0464e2f2c9e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240501 14:11:40.105566 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling FlushMRSOp(b83f4cd48b4b4251a8ac4ddebdb8e4e1): perf score=0.033356
I20240501 14:11:40.117537 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: FlushMRSOp(b83f4cd48b4b4251a8ac4ddebdb8e4e1) complete. Timing: real 0.012s user 0.010s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":1286,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":483,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":160,"threads_started":2,"wal-append.queue_time_us":152}
I20240501 14:11:40.118363 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling UndoDeltaBlockGCOp(b83f4cd48b4b4251a8ac4ddebdb8e4e1): 4921 bytes on disk
I20240501 14:11:40.118731 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: UndoDeltaBlockGCOp(b83f4cd48b4b4251a8ac4ddebdb8e4e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240501 14:11:40.137547 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(b1006490ec2841b2a3365295caaf19e1): perf score=0.033361
I20240501 14:11:40.152176 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(b1006490ec2841b2a3365295caaf19e1) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":178,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":2298,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":493,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":207,"threads_started":2,"wal-append.queue_time_us":186}
I20240501 14:11:40.153277 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(b1006490ec2841b2a3365295caaf19e1): 4834 bytes on disk
I20240501 14:11:40.153656 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(b1006490ec2841b2a3365295caaf19e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
W20240501 14:11:40.165683 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:40.245859 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling FlushMRSOp(b83f4cd48b4b4251a8ac4ddebdb8e4e1): perf score=0.033394
I20240501 14:11:40.258087 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: FlushMRSOp(b83f4cd48b4b4251a8ac4ddebdb8e4e1) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":162,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1330,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":399,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":156}
I20240501 14:11:40.258848 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(b83f4cd48b4b4251a8ac4ddebdb8e4e1): 4921 bytes on disk
I20240501 14:11:40.259200 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(b83f4cd48b4b4251a8ac4ddebdb8e4e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240501 14:11:40.262401 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:40.310761 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:40.401881 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:40.532871 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:11:40.582859 30298 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:40.775193 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:40.817384 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:40.851190 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:40.887158 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:40.978780 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:11:41.027937 30515 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:41.129192 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:41.129992 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:41.169227 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:41.179937 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:41.255568 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:41.386476 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:41.389103 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:41.421265 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:41.441498 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:41.557168 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:41.565245 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:41.602370 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:11:41.613021 30744 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:41.719818 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:41.732213 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:41.735184 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:41.774076 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:41.800663 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:41.826830 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:41.831382 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:41.881134 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:42.043537 30515 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:42.044009 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:42.245494 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:42.260550 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:42.276895 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:42.309140 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:42.363862 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:42.417502 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:42.440958 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:11:42.455545 30515 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:42.480284 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:42.515112 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:42.645403 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:42.718856 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:42.923609 30515 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:42.978003 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:43.061856 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:43.128536 30780 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:43.128664 30780 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:11:43.129237 30780 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:11:43.129518 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:43.129577 473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:11:43.129731 473 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:11:43.130050 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:43.130323 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:43.130406 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:43.130460 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:11:43.130601 30780 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:43.194650 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:43.343932 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:11:43.360673 30744 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:43.383906 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:43.476346 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:11:43.495954 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling FlushMRSOp(9d2c173b2c32494eb16e46a51ca36e31): perf score=0.033370
I20240501 14:11:43.522195 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: FlushMRSOp(9d2c173b2c32494eb16e46a51ca36e31) complete. Timing: real 0.026s user 0.024s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":165,"dirs.queue_time_us":186,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1019,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":364,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":263,"threads_started":3,"wal-append.queue_time_us":174}
I20240501 14:11:43.523254 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling FlushMRSOp(b83f4cd48b4b4251a8ac4ddebdb8e4e1): perf score=0.033360
I20240501 14:11:43.543884 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: FlushMRSOp(b83f4cd48b4b4251a8ac4ddebdb8e4e1) complete. Timing: real 0.020s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":2006,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":495,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":83,"threads_started":1,"wal-append.queue_time_us":151}
I20240501 14:11:43.544814 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling UndoDeltaBlockGCOp(9d2c173b2c32494eb16e46a51ca36e31): 4921 bytes on disk
I20240501 14:11:43.545406 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: UndoDeltaBlockGCOp(9d2c173b2c32494eb16e46a51ca36e31) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 14:11:43.546205 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling UndoDeltaBlockGCOp(b83f4cd48b4b4251a8ac4ddebdb8e4e1): 4921 bytes on disk
I20240501 14:11:43.546656 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: UndoDeltaBlockGCOp(b83f4cd48b4b4251a8ac4ddebdb8e4e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240501 14:11:43.579538 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:43.591260 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:43.602212 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:43.688470 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:43.744868 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:43.780877 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:43.832307 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:11:43.851490 30744 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:44.017930 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:44.027266 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:44.064482 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:44.114027 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240501 14:11:44.114600 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:44.175846 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:44.231586 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:44.266376 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:11:44.293642 30515 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:11:44.299270 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling FlushMRSOp(a7810938b8fb406e9c6ac438f54afd03): perf score=0.033387
I20240501 14:11:44.312778 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: FlushMRSOp(a7810938b8fb406e9c6ac438f54afd03) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1010,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1432,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":437,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":118,"threads_started":2,"wal-append.queue_time_us":109}
I20240501 14:11:44.313524 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling UndoDeltaBlockGCOp(a7810938b8fb406e9c6ac438f54afd03): 4834 bytes on disk
I20240501 14:11:44.313951 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: UndoDeltaBlockGCOp(a7810938b8fb406e9c6ac438f54afd03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240501 14:11:44.314652 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling FlushMRSOp(5aa7bdac7bd1408cbf41d894d40aebd4): perf score=0.033361
I20240501 14:11:44.325803 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: FlushMRSOp(5aa7bdac7bd1408cbf41d894d40aebd4) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1331,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":393,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":128,"threads_started":1,"wal-append.queue_time_us":196}
I20240501 14:11:44.326514 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling UndoDeltaBlockGCOp(5aa7bdac7bd1408cbf41d894d40aebd4): 4581 bytes on disk
I20240501 14:11:44.326839 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: UndoDeltaBlockGCOp(5aa7bdac7bd1408cbf41d894d40aebd4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 14:11:44.327379 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling FlushMRSOp(f26d16499ab54a3e9059f6e4ff041177): perf score=0.033364
W20240501 14:11:44.331089 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:44.338548 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: FlushMRSOp(f26d16499ab54a3e9059f6e4ff041177) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":1210,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":406,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":100,"threads_started":1,"wal-append.queue_time_us":173}
I20240501 14:11:44.339252 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling UndoDeltaBlockGCOp(f26d16499ab54a3e9059f6e4ff041177): 4579 bytes on disk
I20240501 14:11:44.339619 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: UndoDeltaBlockGCOp(f26d16499ab54a3e9059f6e4ff041177) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
W20240501 14:11:44.346128 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:44.353214 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:44.416879 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:44.442507 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:44.450631 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:44.468586 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling FlushMRSOp(f8cc4d3985224178aa008b4c9de5625b): perf score=0.033395
I20240501 14:11:44.492456 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: FlushMRSOp(f8cc4d3985224178aa008b4c9de5625b) complete. Timing: real 0.023s user 0.022s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":167,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":836,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":359,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":236,"threads_started":3,"wal-append.queue_time_us":139}
I20240501 14:11:44.493682 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling UndoDeltaBlockGCOp(f8cc4d3985224178aa008b4c9de5625b): 4921 bytes on disk
I20240501 14:11:44.494143 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: UndoDeltaBlockGCOp(f8cc4d3985224178aa008b4c9de5625b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240501 14:11:44.580466 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:44.590763 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling FlushMRSOp(5b2e30cb9de14fe19273b3cef801cb67): perf score=0.033383
I20240501 14:11:44.607797 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: FlushMRSOp(5b2e30cb9de14fe19273b3cef801cb67) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1252,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":585,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":83,"threads_started":1,"wal-append.queue_time_us":164}
I20240501 14:11:44.608609 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling FlushMRSOp(5d994081c3414f779ba2cae431cb1d97): perf score=0.033376
I20240501 14:11:44.621665 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: FlushMRSOp(5d994081c3414f779ba2cae431cb1d97) complete. Timing: real 0.013s user 0.008s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1268,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":455,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":141}
I20240501 14:11:44.622462 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling FlushMRSOp(96f5e4e25a674a99a6762757f0ab1545): perf score=0.033355
W20240501 14:11:44.624877 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:44.635565 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:11:44.636878 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: FlushMRSOp(96f5e4e25a674a99a6762757f0ab1545) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1293,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":462,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":82,"threads_started":1,"wal-append.queue_time_us":605}
I20240501 14:11:44.637710 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling UndoDeltaBlockGCOp(5d994081c3414f779ba2cae431cb1d97): 4921 bytes on disk
I20240501 14:11:44.638123 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: UndoDeltaBlockGCOp(5d994081c3414f779ba2cae431cb1d97) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240501 14:11:44.638748 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling UndoDeltaBlockGCOp(5b2e30cb9de14fe19273b3cef801cb67): 4834 bytes on disk
I20240501 14:11:44.639072 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: UndoDeltaBlockGCOp(5b2e30cb9de14fe19273b3cef801cb67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240501 14:11:44.639688 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling UndoDeltaBlockGCOp(96f5e4e25a674a99a6762757f0ab1545): 4581 bytes on disk
I20240501 14:11:44.640002 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: UndoDeltaBlockGCOp(96f5e4e25a674a99a6762757f0ab1545) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240501 14:11:44.662806 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 14:11:44.745514 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling FlushMRSOp(ad0dac76fd0c4ae6b8e0549376c98e5d): perf score=0.033401
I20240501 14:11:44.758275 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: FlushMRSOp(ad0dac76fd0c4ae6b8e0549376c98e5d) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1333,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":457,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":136}
I20240501 14:11:44.759259 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling FlushMRSOp(4603d1b1b1d441b1b0be0464e2f2c9e2): perf score=0.033401
I20240501 14:11:44.772487 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: FlushMRSOp(4603d1b1b1d441b1b0be0464e2f2c9e2) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1304,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":436,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":146}
I20240501 14:11:44.773350 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling FlushMRSOp(b1006490ec2841b2a3365295caaf19e1): perf score=0.033389
I20240501 14:11:44.785701 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: FlushMRSOp(b1006490ec2841b2a3365295caaf19e1) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":1311,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":449,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":90,"threads_started":1,"wal-append.queue_time_us":167}
I20240501 14:11:44.786448 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling FlushMRSOp(71241540f585474a8f62da16ed2d9874): perf score=0.033377
I20240501 14:11:44.801753 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: FlushMRSOp(71241540f585474a8f62da16ed2d9874) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1232,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":443,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":2944,"thread_start_us":112,"threads_started":1,"wal-append.queue_time_us":178}
I20240501 14:11:44.802531 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling FlushMRSOp(b67bd0eb45a3486492b75ca3ad3be524): perf score=0.033368
I20240501 14:11:44.813920 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: FlushMRSOp(b67bd0eb45a3486492b75ca3ad3be524) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":1467,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":340,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":90,"threads_started":1,"wal-append.queue_time_us":162}
I20240501 14:11:44.814697 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling UndoDeltaBlockGCOp(71241540f585474a8f62da16ed2d9874): 4921 bytes on disk
I20240501 14:11:44.815057 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: UndoDeltaBlockGCOp(71241540f585474a8f62da16ed2d9874) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240501 14:11:44.815845 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling UndoDeltaBlockGCOp(b1006490ec2841b2a3365295caaf19e1): 4834 bytes on disk
I20240501 14:11:44.816134 30744 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:11:44.816262 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: UndoDeltaBlockGCOp(b1006490ec2841b2a3365295caaf19e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240501 14:11:44.816618 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:44.817013 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling UndoDeltaBlockGCOp(b67bd0eb45a3486492b75ca3ad3be524): 4834 bytes on disk
I20240501 14:11:44.817306 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: UndoDeltaBlockGCOp(b67bd0eb45a3486492b75ca3ad3be524) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4}
I20240501 14:11:44.817842 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling FlushMRSOp(5d994081c3414f779ba2cae431cb1d97): perf score=0.033366
I20240501 14:11:44.829387 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: FlushMRSOp(5d994081c3414f779ba2cae431cb1d97) complete. Timing: real 0.011s user 0.009s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":1254,"drs_written":1,"lbm_read_time_us":32,"lbm_reads_lt_1ms":4,"lbm_write_time_us":410,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":97,"threads_started":1,"wal-append.queue_time_us":177}
I20240501 14:11:44.830070 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling FlushMRSOp(bca09920a7904362bc546916e3f6f5ac): perf score=0.033363
W20240501 14:11:44.830556 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:44.839881 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:11:44.841318 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: FlushMRSOp(bca09920a7904362bc546916e3f6f5ac) complete. Timing: real 0.011s user 0.008s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1089,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":104,"threads_started":1,"wal-append.queue_time_us":252}
I20240501 14:11:44.842144 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling UndoDeltaBlockGCOp(5d994081c3414f779ba2cae431cb1d97): 4921 bytes on disk
I20240501 14:11:44.842442 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: UndoDeltaBlockGCOp(5d994081c3414f779ba2cae431cb1d97) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240501 14:11:44.843011 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling UndoDeltaBlockGCOp(bca09920a7904362bc546916e3f6f5ac): 4921 bytes on disk
I20240501 14:11:44.843303 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: UndoDeltaBlockGCOp(bca09920a7904362bc546916e3f6f5ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240501 14:11:44.843987 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling UndoDeltaBlockGCOp(4603d1b1b1d441b1b0be0464e2f2c9e2): 4581 bytes on disk
I20240501 14:11:44.844257 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: UndoDeltaBlockGCOp(4603d1b1b1d441b1b0be0464e2f2c9e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4}
I20240501 14:11:44.844892 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling UndoDeltaBlockGCOp(ad0dac76fd0c4ae6b8e0549376c98e5d): 4581 bytes on disk
I20240501 14:11:44.845153 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: UndoDeltaBlockGCOp(ad0dac76fd0c4ae6b8e0549376c98e5d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4}
W20240501 14:11:44.899544 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:44.950886 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:44.995878 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:45.028759 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:45.127223 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:45.135262 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:45.230945 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:45.294543 32075 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:11:45.392380 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling FlushMRSOp(c082ccc777ba4f9a937931f707645e54): perf score=0.033381
I20240501 14:11:45.404050 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: FlushMRSOp(c082ccc777ba4f9a937931f707645e54) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":167,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1389,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":437,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":133,"threads_started":2,"wal-append.queue_time_us":121}
I20240501 14:11:45.404886 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling UndoDeltaBlockGCOp(c082ccc777ba4f9a937931f707645e54): 4834 bytes on disk
I20240501 14:11:45.405275 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: UndoDeltaBlockGCOp(c082ccc777ba4f9a937931f707645e54) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240501 14:11:45.433740 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:45.619328 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:45.649852 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:45.662256 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:45.680969 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:45.717262 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:45.827414 32314 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:45.901947 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:11:45.907066 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling FlushMRSOp(8ffdb76b2c5149a18d45bee9581e605a): perf score=0.033368
I20240501 14:11:45.918612 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: FlushMRSOp(8ffdb76b2c5149a18d45bee9581e605a) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1441,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":418,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":133,"threads_started":2,"wal-append.queue_time_us":122}
I20240501 14:11:45.919348 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling UndoDeltaBlockGCOp(8ffdb76b2c5149a18d45bee9581e605a): 4581 bytes on disk
I20240501 14:11:45.919736 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: UndoDeltaBlockGCOp(8ffdb76b2c5149a18d45bee9581e605a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
W20240501 14:11:45.955552 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:45.985633 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:45.989838 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:11:46.099205 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling FlushMRSOp(ea66fe3d0da04d18b961927a9c988a20): perf score=0.033387
I20240501 14:11:46.112663 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: FlushMRSOp(ea66fe3d0da04d18b961927a9c988a20) complete. Timing: real 0.013s user 0.008s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":165,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1564,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":449,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":116}
I20240501 14:11:46.113770 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling UndoDeltaBlockGCOp(ea66fe3d0da04d18b961927a9c988a20): 4581 bytes on disk
I20240501 14:11:46.114145 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: UndoDeltaBlockGCOp(ea66fe3d0da04d18b961927a9c988a20) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240501 14:11:46.150462 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:46.185065 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:46.211728 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:46.227623 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:46.281708 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:46.376523 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:46.425935 32436 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:46.521322 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:46.560982 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:46.579444 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:46.600797 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:46.646343 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:46.660822 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:11:46.672051 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling FlushMRSOp(3c8200d9c20e4140906c71c746badb94): perf score=0.033360
I20240501 14:11:46.687160 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: FlushMRSOp(3c8200d9c20e4140906c71c746badb94) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":394,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1630,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":566,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":127,"threads_started":2,"wal-append.queue_time_us":123}
I20240501 14:11:46.687979 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling UndoDeltaBlockGCOp(3c8200d9c20e4140906c71c746badb94): 4921 bytes on disk
I20240501 14:11:46.688278 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: UndoDeltaBlockGCOp(3c8200d9c20e4140906c71c746badb94) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240501 14:11:46.773039 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:46.774969 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:46.842149 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:46.866868 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling FlushMRSOp(2e68be1521964ba8b864801fdf3b0df3): perf score=0.033355
I20240501 14:11:46.878373 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: FlushMRSOp(2e68be1521964ba8b864801fdf3b0df3) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1362,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":173,"threads_started":2,"wal-append.queue_time_us":157}
I20240501 14:11:46.879134 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling UndoDeltaBlockGCOp(2e68be1521964ba8b864801fdf3b0df3): 4834 bytes on disk
I20240501 14:11:46.879467 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: UndoDeltaBlockGCOp(2e68be1521964ba8b864801fdf3b0df3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240501 14:11:46.880129 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling FlushMRSOp(043344ce36a74a18a0d67ff5bd473e68): perf score=0.033333
I20240501 14:11:46.891667 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: FlushMRSOp(043344ce36a74a18a0d67ff5bd473e68) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1278,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":424,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":136}
I20240501 14:11:46.892575 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling UndoDeltaBlockGCOp(043344ce36a74a18a0d67ff5bd473e68): 4581 bytes on disk
I20240501 14:11:46.893003 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: UndoDeltaBlockGCOp(043344ce36a74a18a0d67ff5bd473e68) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240501 14:11:46.923975 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(9d2c173b2c32494eb16e46a51ca36e31): perf score=0.033360
I20240501 14:11:46.934957 32580 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:11:46.935797 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(9d2c173b2c32494eb16e46a51ca36e31) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1399,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":406,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":127}
I20240501 14:11:46.936684 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(9d2c173b2c32494eb16e46a51ca36e31): 4921 bytes on disk
I20240501 14:11:46.937103 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(9d2c173b2c32494eb16e46a51ca36e31) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240501 14:11:46.972203 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:47.047607 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:47.101349 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:47.144367 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling FlushMRSOp(5b2e30cb9de14fe19273b3cef801cb67): perf score=0.033393
W20240501 14:11:47.151998 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:11:47.156140 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: FlushMRSOp(5b2e30cb9de14fe19273b3cef801cb67) complete. Timing: real 0.012s user 0.008s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1348,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":516,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":140}
I20240501 14:11:47.156965 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling UndoDeltaBlockGCOp(5b2e30cb9de14fe19273b3cef801cb67): 4834 bytes on disk
I20240501 14:11:47.157352 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: UndoDeltaBlockGCOp(5b2e30cb9de14fe19273b3cef801cb67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
W20240501 14:11:47.160229 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:11:47.188453 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(ea66fe3d0da04d18b961927a9c988a20): perf score=0.033361
I20240501 14:11:47.199961 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(ea66fe3d0da04d18b961927a9c988a20) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":1298,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":479,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":137}
I20240501 14:11:47.200736 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(ea66fe3d0da04d18b961927a9c988a20): 4581 bytes on disk
I20240501 14:11:47.201076 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(ea66fe3d0da04d18b961927a9c988a20) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240501 14:11:47.220515 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:47.222733 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:47.265906 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:11:47.369333 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling FlushMRSOp(72ad9c714b2b4fff94063d2838835173): perf score=0.033396
I20240501 14:11:47.381541 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: FlushMRSOp(72ad9c714b2b4fff94063d2838835173) complete. Timing: real 0.012s user 0.011s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":188,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1411,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":400,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":127}
I20240501 14:11:47.382287 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling UndoDeltaBlockGCOp(72ad9c714b2b4fff94063d2838835173): 4834 bytes on disk
I20240501 14:11:47.382642 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: UndoDeltaBlockGCOp(72ad9c714b2b4fff94063d2838835173) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240501 14:11:47.388090 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling FlushMRSOp(ad0dac76fd0c4ae6b8e0549376c98e5d): perf score=0.033365
I20240501 14:11:47.399458 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: FlushMRSOp(ad0dac76fd0c4ae6b8e0549376c98e5d) complete. Timing: real 0.011s user 0.009s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":154,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":1221,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":420,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":141}
I20240501 14:11:47.400236 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling UndoDeltaBlockGCOp(ad0dac76fd0c4ae6b8e0549376c98e5d): 4581 bytes on disk
I20240501 14:11:47.400565 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: UndoDeltaBlockGCOp(ad0dac76fd0c4ae6b8e0549376c98e5d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240501 14:11:47.408600 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling FlushMRSOp(7a542c07a3f841b0a31051a8a37083bf): perf score=0.033393
I20240501 14:11:47.420964 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: FlushMRSOp(7a542c07a3f841b0a31051a8a37083bf) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1350,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":412,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":111,"threads_started":1,"wal-append.queue_time_us":192}
I20240501 14:11:47.421798 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling UndoDeltaBlockGCOp(7a542c07a3f841b0a31051a8a37083bf): 4921 bytes on disk
I20240501 14:11:47.422109 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: UndoDeltaBlockGCOp(7a542c07a3f841b0a31051a8a37083bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 14:11:47.422693 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling FlushMRSOp(72ad9c714b2b4fff94063d2838835173): perf score=0.033385
I20240501 14:11:47.436884 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: FlushMRSOp(72ad9c714b2b4fff94063d2838835173) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":129,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1994,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":470,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":94,"threads_started":1,"wal-append.queue_time_us":223}
I20240501 14:11:47.437777 784 maintenance_manager.cc:419] P a0caf2974041408bb32362bfc4284eaa: Scheduling UndoDeltaBlockGCOp(72ad9c714b2b4fff94063d2838835173): 4834 bytes on disk
I20240501 14:11:47.438145 674 maintenance_manager.cc:643] P a0caf2974041408bb32362bfc4284eaa: UndoDeltaBlockGCOp(72ad9c714b2b4fff94063d2838835173) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240501 14:11:47.451328 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:11:47.525936 32580 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:47.560577 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 14:11:47.565578 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:47.592393 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:47.621101 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:47.651835 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling FlushMRSOp(8f91909d3b1b46e9b858ce5c5b2e7975): perf score=0.033390
I20240501 14:11:47.664793 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: FlushMRSOp(8f91909d3b1b46e9b858ce5c5b2e7975) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1330,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":463,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":63,"threads_started":1,"wal-append.queue_time_us":132}
I20240501 14:11:47.665611 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling UndoDeltaBlockGCOp(8f91909d3b1b46e9b858ce5c5b2e7975): 4834 bytes on disk
I20240501 14:11:47.666029 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: UndoDeltaBlockGCOp(8f91909d3b1b46e9b858ce5c5b2e7975) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240501 14:11:47.708303 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:47.709417 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:47.779850 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling FlushMRSOp(6f8bd03c05d84fdaaf138a6d0608e30e): perf score=0.033376
I20240501 14:11:47.791059 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: FlushMRSOp(6f8bd03c05d84fdaaf138a6d0608e30e) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":177,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1239,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":412,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":153,"threads_started":2,"wal-append.queue_time_us":150}
I20240501 14:11:47.792068 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(6f8bd03c05d84fdaaf138a6d0608e30e): 4581 bytes on disk
I20240501 14:11:47.792480 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(6f8bd03c05d84fdaaf138a6d0608e30e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
W20240501 14:11:47.862705 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:11:47.945744 32726 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:48.024144 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:11:48.043797 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling FlushMRSOp(d265eff200b645bfac8f8906654385f6): perf score=0.033390
I20240501 14:11:48.055817 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: FlushMRSOp(d265eff200b645bfac8f8906654385f6) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":1443,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":406,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":150}
I20240501 14:11:48.056543 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(d265eff200b645bfac8f8906654385f6): 4834 bytes on disk
I20240501 14:11:48.056947 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(d265eff200b645bfac8f8906654385f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240501 14:11:48.067566 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:48.099624 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:48.180917 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:48.182508 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:48.191877 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:48.204617 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling FlushMRSOp(3c8200d9c20e4140906c71c746badb94): perf score=0.033401
I20240501 14:11:48.220216 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: FlushMRSOp(3c8200d9c20e4140906c71c746badb94) complete. Timing: real 0.015s user 0.012s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":164,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1278,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":473,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":161,"threads_started":2,"wal-append.queue_time_us":185}
I20240501 14:11:48.221051 31236 maintenance_manager.cc:419] P 0e4622d80cc94f12b492bdd7597a61e1: Scheduling UndoDeltaBlockGCOp(3c8200d9c20e4140906c71c746badb94): 4921 bytes on disk
I20240501 14:11:48.221419 31126 maintenance_manager.cc:643] P 0e4622d80cc94f12b492bdd7597a61e1: UndoDeltaBlockGCOp(3c8200d9c20e4140906c71c746badb94) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
W20240501 14:11:48.313416 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:11:48.433905 32726 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:11:48.443104 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling FlushMRSOp(4603d1b1b1d441b1b0be0464e2f2c9e2): perf score=0.033341
I20240501 14:11:48.454309 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: FlushMRSOp(4603d1b1b1d441b1b0be0464e2f2c9e2) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1414,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":389,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":139}
I20240501 14:11:48.455010 522 maintenance_manager.cc:419] P b934fd09590a40ada094b9715d990a00: Scheduling UndoDeltaBlockGCOp(4603d1b1b1d441b1b0be0464e2f2c9e2): 4581 bytes on disk
I20240501 14:11:48.455336 412 maintenance_manager.cc:643] P b934fd09590a40ada094b9715d990a00: UndoDeltaBlockGCOp(4603d1b1b1d441b1b0be0464e2f2c9e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240501 14:11:48.592130 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:48.644416 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:48.655445 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:48.669322 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling FlushMRSOp(a7810938b8fb406e9c6ac438f54afd03): perf score=0.033338
I20240501 14:11:48.681756 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: FlushMRSOp(a7810938b8fb406e9c6ac438f54afd03) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":203,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":1334,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":147}
I20240501 14:11:48.682539 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling UndoDeltaBlockGCOp(a7810938b8fb406e9c6ac438f54afd03): 4834 bytes on disk
I20240501 14:11:48.682919 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: UndoDeltaBlockGCOp(a7810938b8fb406e9c6ac438f54afd03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
W20240501 14:11:48.692767 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:48.708621 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:48.765430 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:48.825320 32726 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:48.855324 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:48.937772 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:49.058372 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:49.067344 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:49.138686 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:49.148644 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:49.153983 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 14:11:49.188191 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:49.194926 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:49.197271 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:49.325752 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:11:49.399693 32580 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:49.429872 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:49.464756 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:49.490249 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:49.552358 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:49.624002 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:49.665156 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:11:49.665588 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:11:49.684043 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:49.719045 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:49.770431 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:49.818333 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:11:49.849251 32726 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:49.871480 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:49.876924 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:50.019284 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:50.120558 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:50.237195 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 14:11:50.261461 32726 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:50.283610 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:50.288425 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:50.344686 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:50.345126 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:50.584080 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:50.590770 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:50.687862 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:11:50.716959 32580 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:50.821940 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:50.859889 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:50.893025 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:51.032044 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:51.137864 32726 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:51.139746 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:51.294968 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:51.297173 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:51.335675 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:51.409660 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:51.410558 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:51.497825 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:51.505112 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:51.529389 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:51.577168 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:11:51.609077 32580 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:51.635092 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:51.669736 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:51.887914 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:51.888942 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:51.895042 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:51.948204 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:51.961036 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:51.987543 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:52.001062 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:52.005576 32580 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:52.076524 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:52.165390 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:52.209198 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:52.348899 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:52.365118 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:52.372658 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:52.556818 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:52.577997 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:52.580997 32726 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:52.601401 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:52.691454 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:11:52.692106 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:52.924031 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:52.964803 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:52.972386 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:53.122278 32726 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:53.150312 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:53.191906 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:53.241003 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:53.355157 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:53.386091 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:53.532374 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:53.534013 32726 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:53.554386 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:53.579447 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:53.581482 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:53.632146 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:53.782258 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:53.796908 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:53.893124 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:53.966430 1828 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:53.980440 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:53.992614 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:53.999572 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:54.070322 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:54.099870 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:54.123469 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:54.147652 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:54.267254 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:11:54.297782 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:54.348817 31034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:11:54.349505 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6f8bd03c05d84fdaaf138a6d0608e30e: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:54.424898 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:54.461287 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:54.472470 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:54.521570 1828 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:54.572439 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:54.582653 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:54.601063 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:54.696111 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:54.700372 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:54.721036 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:11:54.728040 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:54.824513 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:54.899796 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:54.899886 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:54.914312 1828 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:54.962960 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:55.038715 27239 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:55.038844 27239 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:11:55.039335 27239 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:11:55.039660 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:11:55.039793 31580 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:11:55.040877 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:55.040982 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:55.041016 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:11:55.041297 27239 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:55.070822 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:55.207383 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:55.254729 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:55.285367 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:55.426376 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:55.478943 1828 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:55.486004 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:55.515462 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:55.560544 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:55.776755 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:55.825109 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:55.863931 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:11:55.868614 2163 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:55.889708 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:55.982270 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:55.989132 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:56.004693 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:56.009048 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:56.236354 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:56.269341 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:56.288987 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:56.375788 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:56.380672 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:11:56.387357 2163 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:56.395270 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:56.510767 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:56.551177 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:56.576371 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:56.669008 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:56.703660 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:56.815275 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:11:56.850813 2576 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:56.894444 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:56.910375 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:56.953524 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:57.094859 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:57.126431 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:57.128672 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:57.248795 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:57.266652 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:57.310051 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:57.328259 2163 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:57.333928 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:57.340998 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:57.477011 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:57.746590 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 14:11:57.747143 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:57.747519 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:57.761580 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:57.878337 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:57.879662 2163 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:57.894976 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:58.031615 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:58.201098 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:58.251895 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:58.278692 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:58.319116 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:58.377941 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:11:58.485335 3083 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:58.551144 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:58.579947 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:58.598384 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:58.655933 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:58.670609 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:58.709466 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:58.739845 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:58.757158 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:58.777397 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:58.781698 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:58.924052 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:11:59.002416 3083 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:59.018251 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:59.126567 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:59.167320 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:59.188267 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:59.228031 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:59.263900 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:59.325794 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:59.327059 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:11:59.327641 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:11:59.395972 2163 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:59.508885 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:59.510867 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:59.542382 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:11:59.591414 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:59.621737 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:59.717607 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:11:59.744544 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:11:59.774340 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:11:59.775110 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:11:59.863283 2163 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:11:59.939530 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:59.952366 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:00.005255 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:00.021772 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:00.101871 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:00.266911 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:12:00.287020 2163 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:00.306892 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:00.379781 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:12:00.452880 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:00.480544 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:00.495672 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:00.531155 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:00.644973 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:12:00.758206 3511 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:00.765699 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:00.879742 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:00.901484 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:00.995839 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:01.066643 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:01.093241 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:01.130776 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:01.138447 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:01.169034 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:01.291885 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:01.307907 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:01.351410 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:12:01.374564 3511 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:01.406376 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:01.472693 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:01.487457 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:01.490705 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:01.549801 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:01.571221 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:01.720430 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:01.782192 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:01.797940 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:01.861344 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:01.890254 3886 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:01.953583 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:01.957548 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:02.013200 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:02.259105 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:02.304651 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:02.357278 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:02.384671 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:02.441418 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:02.460026 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:02.475320 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:12:02.500093 3886 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:12:02.676455 2337 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:02.676602 2337 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:12:02.677171 2337 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:12:02.677647 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:12:02.677817 31581 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:12:02.678409 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:02.678514 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:02.678567 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:12:02.678700 2337 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:02.790136 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:02.818737 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 14:12:02.855549 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:02.900007 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:02.932268 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:03.020169 3511 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:03.022011 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:03.194145 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:03.212213 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:03.313823 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:03.323194 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:03.389429 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:03.393522 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:12:03.461244 3886 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:03.473551 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:03.609019 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:03.623805 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:03.721642 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:03.770331 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:03.771423 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:03.806174 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:03.846040 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:03.912731 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:03.934963 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:03.960779 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:03.990758 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:04.011512 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:04.034282 4391 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:04.043860 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:04.152796 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:04.161185 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:04.268632 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:04.316799 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:04.376776 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 14:12:04.377491 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:04.386583 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:12:04.473891 3886 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:04.506865 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:04.669028 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:04.704921 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:04.776732 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:12:04.792243 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:12:04.794889 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:04.816148 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:04.886382 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:04.991567 3886 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:05.049409 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:05.056252 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:05.173887 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:05.203104 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:05.301887 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:05.328012 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:05.473548 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:05.501899 3886 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:05.548219 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:12:05.550923 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:05.661357 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:05.764321 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:05.826740 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:05.853024 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:05.868546 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:05.949231 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:05.982252 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:12:05.984216 3886 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:06.003737 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:06.065788 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:06.165361 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:06.179041 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:06.292155 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:06.314674 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:06.457989 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:06.473672 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:12:06.502702 4391 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:06.589260 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:06.610540 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:06.616739 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:06.622387 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:06.660456 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:06.741207 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:06.815469 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:06.884644 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:06.907374 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:06.972216 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:07.077515 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:12:07.115968 2337 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:07.116112 2337 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:12:07.116573 2337 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:12:07.116955 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:12:07.117130 31581 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:12:07.117998 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:07.118119 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:07.118166 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:12:07.118305 2337 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:07.121599 5089 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:07.153689 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:07.201143 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:07.214574 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:07.233709 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:07.401582 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:07.506765 4391 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:07.668756 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:07.686244 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:07.707530 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:07.709295 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:07.901526 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 14:12:07.902141 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:07.908435 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:08.032999 5319 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:08.182282 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:08.198771 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:08.230895 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:08.311340 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:08.321339 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:08.364145 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:08.382992 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:08.413120 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:08.450067 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:12:08.460089 5405 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:08.599514 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:08.649674 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:08.725078 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:08.727365 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:08.807405 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:08.811056 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:08.854672 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:08.920418 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:09.049396 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:12:09.063680 5405 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:09.175364 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:09.207980 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:09.225208 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:09.319581 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:09.357434 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:09.447541 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 14:12:09.448275 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:09.479316 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:09.499248 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:09.525715 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:09.557531 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:12:09.668285 5405 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:09.683854 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:09.814150 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:09.818567 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:12:09.838860 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:09.884626 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:12:09.900446 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:09.972394 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:10.202718 5810 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:10.241886 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:10.256081 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:10.310523 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:10.348402 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:10.349160 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:10.490787 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:10.619129 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:12:10.674767 5405 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:10.693517 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:10.718628 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:10.726861 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:12:10.862754 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:10.910264 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:10.922755 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:10.943782 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:10.956475 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:11.012895 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:11.019457 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:11.087882 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:11.205900 6032 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:11.323652 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:11.330026 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:11.389904 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:11.467911 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:11.482714 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:11.503610 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:11.534629 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:11.578954 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:11.582341 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:12:11.639346 5405 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:11.812338 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:11.841226 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:11.841720 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:11.849757 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:11.945020 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:11.954831 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:12.087445 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:12.109068 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:12.141440 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:12:12.223563 6264 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:12.269192 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:12.361531 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:12.466110 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:12.507436 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:12.564894 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:12.569415 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:12.696029 5405 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:12.806623 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:12.814586 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:13.009359 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:13.033113 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:13.038123 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:12:13.120981 5405 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:13.143936 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:13.150959 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:13.265964 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:13.268106 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:13.342136 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:13.352722 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:13.422456 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:13.486477 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:13.492182 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:13.529943 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:13.645347 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:13.673825 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:12:13.744071 6619 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:13.862396 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:13.876029 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:13.889626 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:13.984217 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:14.082471 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:14.184882 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:14.237167 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:14.253676 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:12:14.255368 6619 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:14.314540 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:14.369969 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:14.373154 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:14.395072 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:14.456386 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:12:14.464421 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:14.485746 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:14.522027 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:14.583847 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:14.674001 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:12:14.753540 6619 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:14.764211 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:14.822759 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:14.850675 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:12:14.851325 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:14.854591 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:14.907275 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:15.011768 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:15.017302 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:12:15.118368 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:12:15.202330 6722 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:15.229549 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:15.333581 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:15.405704 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:15.411443 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:15.475605 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:15.590236 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:12:15.787003 7077 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:15.797930 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:15.812312 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:15.875829 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:15.882357 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:15.889092 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:15.914295 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:15.978997 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:16.012104 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:16.130460 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:12:16.134095 2337 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:16.134227 2337 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:12:16.134785 2337 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:12:16.135198 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:12:16.135363 31581 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:12:16.135843 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:16.135936 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:16.135968 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:12:16.136094 2337 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:16.184552 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:16.258409 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:16.387187 7229 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:16.402292 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:16.426391 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:16.477344 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:16.502240 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:16.536470 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:16.766934 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:16.785961 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:16.852759 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:16.866262 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:16.958407 7229 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:16.996215 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:17.010742 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:17.072320 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:17.092351 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:17.102094 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:17.155923 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:17.158131 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:17.244134 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:17.305986 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:17.357658 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:17.385849 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:17.404055 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:17.429742 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:17.468624 7229 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:17.589761 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:17.825822 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:17.844323 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:17.927291 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:17.930259 7229 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:17.948500 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:18.023079 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:18.203347 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 14:12:18.286207 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:18.288219 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:12:18.308041 7675 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:18.308476 7675 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:12:18.309314 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:12:18.309491 31581 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
I20240501 14:12:18.310009 7675 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:12:18.311257 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:18.311362 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:18.311403 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:12:18.311517 7675 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:18.376052 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:18.384649 7229 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:18.397488 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:18.405464 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:18.412532 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:18.434370 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:18.438437 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:18.589524 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:12:18.785979 7229 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:18.806555 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:18.839529 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:18.916893 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:18.946471 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:18.983692 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:19.001706 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:19.020022 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:19.270494 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:19.300702 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:19.307605 7483 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:19.336057 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:19.386755 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:19.415652 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:19.417680 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:19.465307 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:12:19.484134 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:19.545460 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:19.641525 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:19.694293 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:19.748602 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:19.764448 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:19.766249 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:19.798672 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:19.855268 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240501 14:12:19.855831 7483 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:19.855950 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:19.910288 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:19.917770 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:19.921876 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:20.042613 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:12:20.273684 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:20.304322 7483 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:20.315184 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:20.404990 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:20.471125 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:20.514199 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:20.707012 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:20.714473 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:20.735268 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:12:20.815122 8019 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:20.831301 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:20.851053 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:20.851215 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:20.966269 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:21.139855 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:21.207329 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:21.215564 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:21.222978 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:21.234126 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:21.236346 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:12:21.276608 8019 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:21.337735 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:21.352939 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:21.453225 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:21.695154 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:21.747399 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:21.808925 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:12:21.811311 8473 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:21.819195 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:21.873601 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:21.890074 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:21.903785 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:22.021627 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:22.022156 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:22.088424 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:22.097906 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:22.131953 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:22.190382 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:22.196583 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:12:22.242074 8019 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:22.291853 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:22.500276 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:22.506007 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:22.585983 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:22.596906 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:22.667734 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:12:22.679361 8019 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:22.714632 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:22.792083 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:23.015309 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:23.121704 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:23.165979 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:23.219607 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:12:23.224131 8796 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:23.236726 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:23.273523 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:23.356724 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:23.395550 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:23.503525 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:23.553335 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:23.554376 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:23.639134 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:23.645665 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 14:12:23.646317 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:23.733002 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:23.795521 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:23.818970 8796 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:23.896178 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:23.933327 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:24.076998 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:24.114579 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:24.147991 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:12:24.240199 8796 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:24.247438 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:24.329210 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:24.429067 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:24.452667 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:24.460227 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:24.494391 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:12:24.495030 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:24.509316 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:24.529498 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:24.595185 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:24.707113 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:24.742424 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:12:24.787639 9172 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:24.800629 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:24.811012 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:24.860718 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:24.877527 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:12:24.890794 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:24.949710 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:25.040481 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:25.045367 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:12:25.046101 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:25.285853 8796 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:25.344796 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:25.449137 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:25.559348 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:25.612668 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:25.653952 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:25.708539 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:25.757524 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:25.820508 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:25.823159 8796 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:25.920198 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:26.008478 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:26.024678 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:26.075582 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:26.110430 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:26.183871 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:12:26.216929 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:26.296654 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:26.308111 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:26.376983 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:12:26.444197 9538 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:26.517164 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:26.535821 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:26.566931 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:26.704638 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:26.763784 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:26.909490 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:26.915278 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:27.003535 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:27.006695 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:27.016258 9538 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:27.096885 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:27.117713 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:27.142871 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:27.193634 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:27.196988 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:27.226292 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:27.283449 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:27.461344 9538 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:27.477072 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:27.489791 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:27.499689 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:27.560211 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:27.653059 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:27.731209 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:27.813959 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:27.818106 8544 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:27.818239 8544 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:12:27.818776 8544 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:12:27.819303 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:12:27.819468 31580 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:12:27.820000 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:27.820423 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:27.820487 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:12:27.820621 8544 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:27.874331 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:12:27.942771 9783 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:27.962054 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:28.133001 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:28.209616 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:28.283157 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:28.363324 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:28.363523 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:28.379480 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:28.394690 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:28.431568 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:12:28.441797 9783 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:28.637009 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:28.747385 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:28.751010 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:28.784893 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:28.806767 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240501 14:12:28.851603 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:28.865964 9783 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:28.929898 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:28.996760 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:29.003137 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:29.109807 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:29.175778 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:29.277956 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:29.306800 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:12:29.308368 9681 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:29.318781 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:29.320698 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:29.352169 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:29.374382 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:29.476320 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:29.539613 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 14:12:29.598943 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:29.683526 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:29.729595 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:29.740191 9783 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:29.815488 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:29.956328 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:29.974085 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:12:29.984905 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:30.065234 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:30.077601 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:12:30.170136 10302 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:30.182353 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:30.225168 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:12:30.225692 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:30.267203 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:30.282485 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:30.308573 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:30.542182 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:30.562793 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:12:30.625975 9783 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:30.708737 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:30.732582 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:30.889546 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:31.005685 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:31.009774 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:31.054252 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:31.083367 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:31.146219 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:31.154491 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:31.230129 10302 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:31.324159 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:31.335178 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:31.348426 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:31.353125 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:31.366868 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:31.438513 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:31.451072 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:31.607990 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:31.657721 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:31.725008 10636 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:31.743322 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:31.784978 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:31.836498 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:31.986732 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:32.021538 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:32.044226 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:32.118929 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:32.177592 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:32.184357 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:32.224980 10302 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:32.244638 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:32.264412 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:32.376945 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:32.452087 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:32.490415 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:32.573676 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:32.602304 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:32.637115 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:12:32.660207 10302 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:32.713503 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:32.745070 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:32.860553 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:32.895411 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:12:33.074131 10636 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:33.106596 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:33.221239 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:33.279353 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:33.392850 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:33.568645 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:33.581547 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:33.601634 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:33.648483 11191 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:33.731786 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:33.754616 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:33.782021 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:33.858726 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:33.873796 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:33.881814 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:12:33.918234 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:33.943411 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:33.989131 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:12:34.030995 10636 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:34.139590 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:34.146297 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:34.182835 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:34.292416 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:34.353232 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:34.362058 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:34.448096 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:34.506250 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:34.512688 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:12:34.597868 11404 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:34.706581 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:34.712090 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:12:34.712687 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:34.712778 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:34.729872 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:34.744733 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:34.816167 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:34.831496 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:34.889827 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:34.921571 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:35.001163 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:12:35.041554 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:35.130210 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:35.174454 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:35.221086 11547 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:35.283772 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:35.286684 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:35.438441 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:35.582182 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:12:35.661314 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:35.679194 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:35.801393 11547 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:35.856676 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:35.860886 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:35.897997 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:35.993994 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:36.119776 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:36.214392 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:36.273007 11699 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:36.303273 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:36.311304 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:36.369720 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:36.398309 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:36.423799 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:36.429800 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:36.510669 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:36.637352 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:36.686573 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:36.702334 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:36.723520 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:36.770848 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:12:36.835659 11933 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:36.837271 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:36.844298 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:36.892963 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:36.941951 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:36.962191 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:36.988621 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:37.126746 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:37.186832 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:37.218956 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:37.263710 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:37.395679 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:12:37.411245 12055 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:37.466580 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:37.478569 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:37.508414 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:37.552042 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:37.663255 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:37.728967 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:37.740103 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:37.941318 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:38.003384 12208 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:38.076311 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:38.274410 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:38.315457 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:38.391651 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:38.467711 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:12:38.531742 12208 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:38.628121 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:38.638660 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:38.705824 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:38.791874 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:38.830529 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:38.878268 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:38.896677 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:38.921957 12325 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:38.940310 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:39.148557 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:39.192739 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:39.196367 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:39.252517 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:12:39.304920 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:39.317459 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:39.320637 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:39.360201 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:39.389525 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:12:39.393234 8544 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:39.393357 8544 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:12:39.393875 8544 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:12:39.394204 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:12:39.394359 31580 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:12:39.395020 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:39.395119 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:39.395167 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:12:39.395298 8544 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:39.410728 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:39.427204 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:39.463451 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:12:39.475890 12208 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:39.529192 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:39.541610 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:39.733160 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:39.791932 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:39.871026 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:39.877913 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:12:39.878423 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:39.885957 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:39.966243 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:39.966727 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:40.007195 12208 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:40.019559 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:40.177814 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:12:40.178460 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:40.346535 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:40.366133 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:40.507128 12208 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:40.550420 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:40.564985 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:40.572062 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:40.681165 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:40.779436 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:40.798228 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:12:40.931768 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:40.980269 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:41.043932 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:41.077361 12906 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:41.079269 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:41.175993 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:41.275817 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:41.312885 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:41.389467 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:41.401032 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:41.450482 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:41.534430 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:41.540060 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:41.555078 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:41.615123 12906 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:41.676677 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:41.705018 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:41.759382 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:41.856371 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:41.883582 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:41.892416 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:41.903637 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:41.954450 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:41.976233 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:42.019294 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:42.030898 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:12:42.091744 12906 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:42.299705 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:42.303503 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:42.328133 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:42.444262 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:42.454628 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:42.482609 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:42.510252 13049 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:42.563632 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:12:42.695472 8544 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:42.695577 8544 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:12:42.695984 8544 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:12:42.696301 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:12:42.696456 31581 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:12:42.697386 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:42.697492 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:42.697535 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:12:42.697660 8544 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:42.737183 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:42.869697 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:42.911331 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:42.921334 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:42.995819 13049 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:43.068007 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:43.098439 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:43.265082 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:43.298003 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:43.372040 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:43.441601 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:43.450927 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:43.471275 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:43.494884 12906 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:43.669968 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:43.691881 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:43.710690 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:43.764942 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:43.890185 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:43.923444 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:43.929718 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:43.981662 13049 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:43.982470 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:44.030673 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:44.074306 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:44.127180 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:44.129964 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:44.257922 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:44.315433 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 14:12:44.337105 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:44.347584 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:44.367305 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:44.378162 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:44.419741 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:44.428426 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:12:44.598942 13049 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:44.611505 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:44.627101 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:44.674343 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:44.759680 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:44.796628 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:44.846825 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:44.946627 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 14:12:44.950877 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:45.082916 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:12:45.150940 13580 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:45.153030 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:45.177362 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:45.231035 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:12:45.273607 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:45.351647 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:45.495056 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:45.540175 13580 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:45.561347 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:45.670850 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:45.679924 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:45.743700 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:45.756984 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:45.885377 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:45.929540 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:46.052052 13580 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:46.077225 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:12:46.108294 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:46.125764 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:46.144210 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:46.156899 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:46.247849 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:46.323711 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:46.415822 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:46.450723 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:46.457129 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:46.457737 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:12:46.536245 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:46.538408 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:46.675513 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:46.722584 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:46.731410 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:46.739847 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:46.835991 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:46.945281 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:46.951656 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:47.001833 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:47.037271 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:47.100843 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:47.108043 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:12:47.147089 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:47.225526 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:47.253386 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:47.271157 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:47.366662 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:47.566249 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:47.581969 14315 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:47.591135 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:47.644702 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:47.821794 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:47.875406 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:47.959720 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:12:47.989964 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:48.039018 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:48.080642 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:48.119318 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:48.359884 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:48.406567 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:48.418656 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:48.449040 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:48.497507 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:48.505529 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:48.626344 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:48.646322 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:48.673794 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:48.724057 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:48.778528 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:48.912962 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:48.924604 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:48.928328 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:48.967482 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:48.990375 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:48.995370 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:49.155362 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:49.276343 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:49.338320 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:49.403055 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:49.410239 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:49.453101 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:49.467242 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:49.492601 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:49.501931 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:49.540733 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:49.554584 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:49.654067 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:49.701042 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:12:49.756600 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:49.820391 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:49.893767 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:49.938392 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:49.967684 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:12:49.968271 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:50.043527 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:50.061049 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:50.140501 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:50.196440 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:50.232860 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 14:12:50.323586 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:50.361268 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:50.528939 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:50.576762 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:50.590171 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:50.660938 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:50.724798 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:50.743928 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:12:50.794152 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:50.882519 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:50.903996 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:50.980021 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:50.982860 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:50.985002 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:51.079742 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:12:51.080132 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:51.139652 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:51.203760 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:51.338661 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:51.371613 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:51.393297 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:51.410853 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:51.411310 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:51.461478 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:51.543843 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:51.746831 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:51.746865 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:51.921408 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:51.924990 15407 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:51.952883 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:51.968101 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:52.013801 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:52.041064 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:52.051299 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:52.096509 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:52.119194 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:52.143826 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:52.230067 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:52.325196 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:52.535540 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:52.540751 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:52.571166 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:52.606314 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:52.654139 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:52.801610 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:52.884588 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:12:53.004619 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:53.024693 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:53.091192 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:53.094444 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:53.136746 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:53.151852 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:53.370931 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:53.386595 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:53.404078 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:53.426385 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:53.514580 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:53.542659 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:53.583268 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:53.657099 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:53.667016 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:53.676331 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:53.714870 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:53.841179 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:53.940991 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:53.951350 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:53.980042 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:54.004932 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:54.032029 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:54.125144 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:54.163440 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:54.363075 31034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:12:54.363878 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6f8bd03c05d84fdaaf138a6d0608e30e: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:54.412313 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:54.454438 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:54.461071 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:54.472929 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:54.503225 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:54.526005 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:54.530965 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:54.581204 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:54.593129 16017 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:54.614140 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:54.640717 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:54.852627 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:54.969281 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:12:55.024338 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:55.039088 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:55.045583 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:12:55.058759 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:55.062711 14060 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:55.134949 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:55.297309 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 14:12:55.440111 16017 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:55.460533 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:55.465041 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:55.483341 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:55.526690 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:55.548086 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:55.700489 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:55.705786 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:55.788452 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:55.878839 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:55.879108 16017 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:55.917116 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:55.949541 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:55.959882 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:56.000998 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:56.128741 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:56.157102 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:56.172946 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:56.244273 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:56.296378 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:12:56.309456 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:56.361371 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:56.413900 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:56.425721 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:56.429319 16435 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:56.465415 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:56.622555 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:56.746941 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:12:56.747612 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:56.765405 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:12:56.809415 16017 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:56.823734 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:56.860276 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:56.864691 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:56.877962 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:56.891753 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:57.032110 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:57.112023 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:57.190163 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:57.241823 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:57.272771 16017 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:57.318169 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:57.335453 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:57.360942 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:57.474908 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:57.656810 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:12:57.785698 16435 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:57.892668 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:57.923362 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:57.951825 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:58.062541 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:58.083173 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:58.348260 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:58.351397 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:12:58.384058 16435 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:58.461045 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:58.501246 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:58.516799 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:58.517340 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:58.576481 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:58.655059 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:58.669337 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:58.712145 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:58.752131 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:58.833014 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:58.856346 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:58.960584 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:58.993804 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:58.995898 17030 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:59.047555 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:59.108485 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:59.116767 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:59.244103 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:59.265560 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:59.393970 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:59.416177 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:12:59.427983 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:59.436966 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:59.445341 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:12:59.457134 17030 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:12:59.514977 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:59.556241 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:59.651242 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:59.660239 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:59.747017 8544 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:59.747169 8544 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:12:59.747767 8544 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:12:59.748101 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:12:59.748291 31581 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:12:59.749161 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:59.749264 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:12:59.749297 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:12:59.749428 8544 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:59.830361 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:59.969033 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:12:59.975627 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:12:59.996003 17030 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:00.027554 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:00.153288 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:13:00.159365 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:13:00.253837 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:00.279567 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:00.396660 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 14:13:00.429754 17263 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:00.518996 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:00.623463 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:00.705332 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:00.727437 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:00.838960 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:00.892190 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:00.920869 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:00.941135 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:00.968756 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:13:01.010794 17263 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:01.123775 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:01.146214 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:01.167541 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:01.271394 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:01.313266 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:01.324002 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:01.365728 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:01.372958 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:01.414397 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:01.437281 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:01.438253 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:01.542739 17263 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:01.551061 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:01.572855 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:01.683368 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:01.765053 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:13:01.765666 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:01.766978 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:01.824328 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:01.831326 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:01.876641 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:01.952848 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:02.002620 17263 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:02.126948 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:02.147670 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:02.162139 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:02.217566 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:02.246136 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:02.335919 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:02.417847 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:02.601792 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:02.611325 17874 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:02.612479 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:02.716542 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:02.733043 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:02.804720 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:03.045058 17263 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:03.143412 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:03.155331 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:03.184654 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:03.193030 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:03.289781 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:03.326812 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:13:03.444051 17263 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:03.473747 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:03.511767 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:03.711661 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:03.720644 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:03.733359 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:03.739897 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:03.787407 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:03.805512 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:03.809324 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:03.838754 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:13:03.851135 17874 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:03.887696 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:03.888489 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:03.906352 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:04.192489 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:04.204392 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:04.248310 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:04.263918 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:04.293241 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:13:04.293682 17874 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:04.324335 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:04.395090 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:04.478297 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:04.598394 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:04.716446 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:13:04.770102 17874 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:04.827379 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:04.839910 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:04.883626 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:04.891598 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:04.951826 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:04.975387 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:05.077962 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:05.208413 17263 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:05.262240 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:13:05.362095 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:05.379608 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:05.391911 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:05.417722 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:13:05.463446 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:13:05.586117 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:05.606055 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:05.693995 17263 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:05.857486 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:05.921541 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:05.955035 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:13:05.980618 17247 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:13:05.980748 17247 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:13:05.981478 17247 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:13:05.981911 472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:13:05.982084 472 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:13:05.982875 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:05.983176 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:05.983261 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:13:05.983405 17247 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:05.987061 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:05.999724 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:06.014389 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:06.083613 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:06.084475 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:06.143431 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:06.148900 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:06.260423 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:13:06.273998 18725 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:06.282382 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:06.417225 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:06.430505 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:06.436966 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:06.466405 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:06.514254 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:13:06.575667 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:06.659888 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:06.718096 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:06.724325 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:06.750403 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:13:06.789467 18849 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:06.859762 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:06.870204 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:06.930857 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:07.030453 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:07.175444 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:07.178041 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:07.244127 18725 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:07.252210 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:07.295107 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:07.330050 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:07.364411 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:07.501425 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:07.538756 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:07.640767 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:07.708462 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:07.792233 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:07.804597 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:07.855362 19084 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:08.154358 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:08.176008 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:08.248358 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:08.280464 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:08.300948 19084 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:08.392124 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:08.398190 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:08.446441 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:08.522756 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:08.568465 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:08.579623 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:08.582749 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:08.638521 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:08.727914 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:08.756579 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:13:08.799571 19084 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:08.825556 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:08.861016 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:08.908166 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:08.940645 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:08.971900 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:09.010690 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:09.102044 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:09.187104 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:09.276726 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:09.340549 19445 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:09.396276 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:09.514426 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:09.541447 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:09.592151 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:09.609570 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:09.656390 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:09.666913 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:09.709441 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:09.709527 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:09.757969 19084 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:09.994930 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:10.017530 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:10.043277 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:10.056020 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:10.208673 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:10.256435 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:10.289788 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240501 14:13:10.369748 19084 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:10.463951 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:13:10.500063 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:10.510417 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:10.599355 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:10.655865 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:10.663154 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:10.673528 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:13:10.809041 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:13:10.892081 19084 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:10.950069 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:10.977996 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:11.036332 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:11.047518 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:11.074530 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:11.111267 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:11.162868 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:11.256012 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:11.275271 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:13:11.322638 19084 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:11.390043 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:11.428359 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:11.471129 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:11.473459 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:11.488497 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:11.490006 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:11.499804 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:11.521808 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:11.548480 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:11.647492 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:11.677873 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:11.859694 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:11.872328 19084 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:11.955539 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:11.991525 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:12.022477 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:12.033617 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:12.042953 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:12.088567 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:12.170384 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:12.334817 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:12.364378 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:12.388480 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:12.427309 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:12.459359 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:12.461350 20160 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:12.531881 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:12.598974 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:12.881353 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:12.913007 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:12.979212 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:13.011348 20292 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:13.020951 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:13.113979 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:13.298840 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:13.367664 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:13.402483 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:13.438038 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:13.443519 20292 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:13.473875 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:13.490453 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:13.512210 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:13.551945 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:13.634709 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:13.699889 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:13.726209 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:13.751953 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:13.819706 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:13.820485 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:13.841770 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:13.851431 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:13.874256 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:14.032897 20517 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:14.070761 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:14.115839 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:14.233829 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:14.247843 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:14.395511 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:14.410697 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:14.478745 20517 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:14.550725 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:14.583431 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:14.687582 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:14.709940 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:14.854207 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:14.860525 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:14.892982 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:14.941865 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:14.962016 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:15.020354 20292 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:15.092067 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:15.094656 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:15.371516 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:13:15.372138 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:15.404654 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:15.444504 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:15.516573 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:15.521364 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:13:15.532744 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:15.591374 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:13:15.601442 20292 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:15.663270 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:15.678025 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:15.713356 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:13:15.885393 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:15.908589 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:15.926532 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:16.017940 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:16.084983 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:16.101619 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:13:16.126930 20292 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:13:16.180137 20736 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:13:16.180289 20736 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:13:16.181300 472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:13:16.180824 20736 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:13:16.181447 472 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:13:16.182148 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:16.182260 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:16.182312 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:13:16.182448 20736 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:16.204602 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:16.264456 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:16.285704 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:16.384888 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:16.411021 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:16.411098 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:16.469899 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:16.479887 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:16.496455 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:16.500934 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:16.526083 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:13:16.526687 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:16.610085 21005 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:16.902666 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:16.903084 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:16.935500 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:16.976339 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:17.017479 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:17.035406 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:17.039906 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:17.129985 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:17.130841 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:17.179917 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:13:17.219586 21262 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:17.377781 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:17.468159 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:17.493790 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:17.540500 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:17.616863 21005 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:17.717980 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:17.905678 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:17.962015 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:18.007119 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:18.039072 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:18.132977 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:13:18.134971 21005 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:18.135190 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:18.229398 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:18.237557 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:18.441381 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:18.452723 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:18.501142 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:18.506523 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:18.682672 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:13:18.692041 21005 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:18.785856 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:18.814910 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:18.865393 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:18.888362 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:18.890422 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:18.895197 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:18.914392 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:18.928342 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:18.943926 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:18.977033 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:19.065770 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:19.131505 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:13:19.223578 21728 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:19.224699 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:19.296492 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:19.385044 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:19.412146 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:19.435421 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:19.513763 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:19.519887 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:19.610967 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:19.612056 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:13:19.623980 21728 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:19.904596 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:19.913450 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:19.977752 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:19.980091 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:19.998946 21728 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:20.325114 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:20.367252 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:20.396301 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 14:13:20.396997 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:20.420228 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:20.486258 21005 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:20.502358 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:20.555810 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:13:20.556385 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:20.570173 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:20.624617 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:20.629276 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:20.715451 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:20.885437 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:20.898593 21005 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:20.922895 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:20.928386 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:13:21.110826 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:21.158870 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:21.243891 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:21.255419 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:21.271885 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:21.348861 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:21.348954 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:21.389132 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:21.522253 22267 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:21.530750 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:21.531735 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:21.547118 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:21.564397 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:21.585851 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:21.590965 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:13:21.591585 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:21.630148 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:21.666651 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:21.695196 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:21.737074 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:21.744221 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:21.775889 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:21.829334 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:21.959028 22267 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:22.033068 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:22.093129 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:22.196131 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:22.323501 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:22.324920 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:22.326455 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:22.458786 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:22.493778 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:22.574687 22267 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:22.725821 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:22.743659 21727 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:13:22.743808 21727 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:13:22.744307 21727 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:13:22.744706 472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:13:22.745040 472 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:13:22.745713 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:22.745826 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:22.745880 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:13:22.746034 21727 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:22.865072 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:22.896173 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:22.903541 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:22.971984 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:22.996237 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:23.023301 22267 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:23.088069 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:23.101325 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:23.246666 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:23.275394 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:23.446012 22267 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:23.459134 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:23.461982 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:23.467506 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:23.607200 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:23.657827 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:23.715006 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:23.849587 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:23.850715 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:23.910313 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:23.923708 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:23.955432 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:23.960364 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:23.983165 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:23.997368 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:24.019713 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:24.032174 22848 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:24.050633 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:24.073647 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:24.101620 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:24.127067 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:24.317445 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:24.346843 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:24.410820 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:24.449688 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:24.522148 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:24.563937 22980 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:24.805807 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:24.840096 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:24.867797 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:24.974814 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:24.985430 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:13:25.045377 22980 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:25.247256 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:25.323834 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:25.339473 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:25.414357 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:13:25.414935 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:25.420394 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:25.556165 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240501 14:13:25.574671 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:13:25.598155 23213 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:25.683864 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:25.695814 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:25.709465 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:25.711596 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:25.749079 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:25.872949 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:25.929502 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240501 14:13:26.007114 23213 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:26.013032 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:26.064940 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:26.099311 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:26.161315 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:26.212109 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:26.236455 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:26.262090 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:26.262382 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:26.294056 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:26.321790 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:26.397817 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:26.523581 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:26.578624 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:13:26.589653 23437 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:26.592650 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:26.652665 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:26.684391 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:26.696785 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:13:26.697566 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:26.757942 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:26.772732 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:26.779500 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:26.932960 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:27.051626 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:13:27.105895 23538 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:27.198276 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:27.212487 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:27.238099 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:27.283124 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:27.441123 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:27.669366 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:27.693972 23690 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:27.744086 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:27.781883 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:27.829314 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:27.837865 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:27.925762 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:28.088868 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:28.236083 23817 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:28.239203 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:28.240818 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:28.254010 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:28.256711 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:28.301064 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:28.365533 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:28.446614 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:28.501888 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:28.554623 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:28.575877 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:28.590152 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:28.724756 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:28.771301 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:28.777534 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:28.787315 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:13:28.826082 23817 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:28.851862 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:28.886905 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:28.921974 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:28.968698 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:29.087564 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:29.165593 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:29.177268 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:13:29.211804 23690 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:29.245026 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:29.258261 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:29.278187 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:29.372284 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:29.404098 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:29.598850 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:29.603580 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:29.750499 23690 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:29.859741 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:29.890630 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:29.953797 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:30.050594 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:30.166344 24171 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:30.290316 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:30.325906 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:30.327139 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:30.357518 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:30.459646 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:13:30.460336 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:30.589560 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:30.606768 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:30.611254 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:30.644061 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:13:30.644793 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:13:30.681541 24170 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:30.740079 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:30.763540 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:30.776767 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:30.868177 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:30.918329 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:30.924464 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:30.965931 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:30.976758 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:30.986359 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:31.131350 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:31.146278 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:31.202284 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:31.232765 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:31.273516 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 14:13:31.276662 24170 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:31.282733 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:31.402621 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:31.442536 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:31.620012 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:31.622309 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:31.637986 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:13:31.701874 24527 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:31.714430 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:31.795581 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:13:31.796211 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:31.812120 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:31.869120 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:31.892050 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:31.913079 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:32.004215 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:32.079243 24170 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:32.263875 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:32.307289 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:32.319469 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:32.326179 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:13:32.479267 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:32.590102 24170 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:32.625439 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:32.770210 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:32.786880 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:32.857352 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:32.881992 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:32.971419 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:32.989436 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:13:33.060979 24527 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:33.111009 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:33.149384 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:33.231329 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:33.287261 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:33.382421 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:33.397414 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:33.427812 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:33.440668 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:33.446154 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:33.462846 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:33.585340 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:13:33.614977 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:33.649928 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:33.661422 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:33.773548 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:33.836220 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:33.856873 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:33.957535 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:34.004038 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:34.008261 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:34.018289 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:34.068782 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:34.119781 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:34.226461 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:34.253978 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:34.347183 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:34.397686 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:34.517002 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:34.574366 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:34.611961 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:34.748317 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:34.824239 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:34.973047 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:34.999655 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:35.002732 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:35.088013 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:35.197260 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:35.244172 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:35.343152 25105 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:13:35.343300 25105 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:13:35.343843 25105 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:13:35.344211 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:13:35.344358 31581 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:13:35.345043 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:35.345137 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:35.345182 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:13:35.345321 25105 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:35.419344 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:35.431064 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:35.439339 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:35.470472 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:35.515743 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:35.536254 25296 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:35.579309 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:13:35.609536 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:35.662293 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:35.703097 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:13:35.703888 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:35.756405 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:35.791079 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:35.922899 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:35.985697 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:35.991549 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:13:36.008908 25296 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:36.032155 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:36.050323 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:36.119204 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:36.150472 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:36.171733 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:36.254850 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:36.441174 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:36.449371 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240501 14:13:36.454443 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:36.462445 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:36.479998 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:36.490161 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:13:36.494668 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:36.528273 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:36.529521 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:36.616950 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:36.618460 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:36.668561 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:36.835966 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:36.836699 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:36.901327 25296 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:36.923040 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:36.945436 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:37.088949 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:37.092486 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:37.393306 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:37.414841 25296 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:37.443328 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:37.478482 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:37.590251 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:37.640138 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:37.671453 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:37.842077 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:37.879204 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:37.911849 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:37.975543 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:37.979173 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:38.097568 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:38.158789 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:38.178061 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:38.201561 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:38.292889 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:38.316089 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:13:38.330384 25296 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:38.389802 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:38.430465 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:38.499760 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:38.548805 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:38.594362 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:38.702149 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:38.755295 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:38.771876 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:38.793288 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:38.818871 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:38.873080 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:13:38.914541 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:38.971689 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:38.991562 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:39.069962 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:39.120922 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:39.181085 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:39.312006 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:39.326488 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:39.330283 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:39.347126 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:39.413267 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:39.473222 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:39.655961 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:39.714753 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:39.737849 25296 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:39.794690 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:39.969439 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:13:39.975174 26518 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:13:39.975315 26518 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:13:39.976004 26518 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:13:39.976329 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:13:39.976505 31581 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:13:39.977396 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:39.977658 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:39.977720 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:13:39.977929 26518 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:39.978184 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:40.219197 25296 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:40.219534 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:40.234531 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:40.360332 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:40.423681 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:40.579301 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:13:40.638459 25296 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:40.643151 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:40.707274 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 14:13:40.717916 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:40.763360 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:40.799283 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:40.850090 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:13:40.850682 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:40.871868 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:40.889129 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:40.966890 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:40.977847 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:41.014701 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:41.025045 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:13:41.039316 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:41.125444 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:41.185101 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:41.243708 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:41.244160 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:41.292070 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:41.345628 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:41.357345 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:41.380343 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:41.520145 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:13:41.521924 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:41.609923 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:41.653556 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 14:13:41.654191 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:41.765933 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:41.797663 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:41.822942 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:41.836061 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240501 14:13:41.836678 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:41.865569 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:41.902702 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:41.932185 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:13:42.042330 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:42.231446 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:42.374838 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:42.391103 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:42.405696 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:42.483532 27001 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:42.614320 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:42.656152 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:42.817587 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:42.826504 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:42.869482 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:42.896520 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:42.908154 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:43.098848 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:43.122442 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:43.157440 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:43.303045 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:43.335804 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:43.348470 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:43.375615 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:43.386875 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:43.420594 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:43.423776 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:43.465763 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:43.559370 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:43.674798 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:43.683413 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:43.727342 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:43.753472 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:43.780891 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:43.812135 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:43.826030 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:43.842185 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:13:43.910799 26518 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:13:43.910930 26518 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:13:43.911398 26518 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:13:43.911844 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:13:43.911998 31581 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:13:43.912750 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:43.912895 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:43.912947 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:13:43.913066 26518 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:43.933264 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:43.962983 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:43.987294 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:43.991925 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:44.233630 27001 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:44.296937 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:44.337543 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:44.384363 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:44.406814 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:44.416489 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:44.508169 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:44.557163 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:44.607395 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:13:44.637231 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:44.764518 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:44.824041 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:44.828603 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:45.021860 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:45.147091 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:45.191642 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:45.193339 27706 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:45.360479 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:45.398425 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:45.437916 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:45.451613 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:45.643766 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:45.687505 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:45.705068 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:13:45.717976 25044 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:45.790026 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:45.810484 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:13:45.851001 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:45.939169 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:45.950618 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:45.989997 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:45.997119 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:13:45.997730 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:46.023553 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:46.095459 27706 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:46.115891 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:46.151122 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:46.266863 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:46.269559 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:46.269835 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:46.424814 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:46.443544 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:46.490012 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:46.507776 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:46.508147 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:46.537638 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:46.637246 27706 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:46.699102 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:46.734629 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:46.759127 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:46.836491 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:13:46.837105 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:46.982199 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 14:13:46.982847 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:46.984335 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:46.984647 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:13:47.083909 27706 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:47.141007 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:47.182147 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:47.254135 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:47.358722 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:47.566679 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:47.571609 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:47.574474 27706 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:47.711527 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:47.748095 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:47.758105 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:47.758198 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:47.777149 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:47.971994 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:48.112378 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:13:48.129827 28387 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:48.132759 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:48.154412 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:48.154850 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:48.310151 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:48.334935 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:48.398007 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:48.439637 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:48.448786 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:48.465958 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:48.505511 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:48.517249 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:13:48.532999 28387 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:48.572170 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:48.579169 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:48.792881 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:48.797478 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:48.815217 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:48.836652 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:49.042867 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:49.052395 28387 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:49.061614 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:49.070624 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:49.076421 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:49.200084 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:49.229756 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:49.255542 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:49.307469 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:49.316869 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:49.584451 28589 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:49.587563 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:49.628736 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:49.634730 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:49.646134 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:49.754540 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:49.814440 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:49.980012 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:50.060997 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:50.075122 28589 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:50.105882 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:50.188692 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:50.374926 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:50.386447 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:50.424872 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:13:50.451668 28387 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:50.566093 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:50.595065 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:50.602110 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:50.654844 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:50.710333 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:50.820780 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:50.836355 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:13:50.838703 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:50.904469 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:50.924963 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:13:51.034862 29025 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:51.058763 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:51.082118 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:51.083567 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:13:51.084190 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:51.147089 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:51.180370 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:51.185819 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:51.413097 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:51.422189 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:13:51.522167 29025 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:51.531322 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:51.605340 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:51.725752 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:51.743260 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:51.745765 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:51.749513 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:51.770774 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:51.820284 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:51.937776 28872 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:13:51.937925 28872 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:13:51.938535 28872 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:13:51.938894 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 14:13:51.939193 473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:13:51.939339 473 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:13:51.939947 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:51.940045 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:51.940086 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:13:51.940217 28872 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:13:51.961004 29025 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:52.060381 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:52.091506 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:52.128870 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:13:52.151460 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:52.153857 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:52.195389 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:52.259534 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:52.280198 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:52.534797 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:13:52.577034 28387 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:52.591071 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:52.744436 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:52.767338 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:52.768934 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:52.782555 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:52.971159 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:13:53.043949 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:13:53.076483 29025 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:53.117274 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:53.193706 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:53.214402 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:53.247890 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:53.263406 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:53.276060 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:53.285877 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:53.322278 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:53.496628 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:53.549978 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:53.601699 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:13:53.628299 29639 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:53.641889 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:53.649248 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:53.760408 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:53.760419 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:53.881702 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:54.015625 29639 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:54.016292 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:54.038069 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:54.151391 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:54.200361 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:54.232537 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:54.264467 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:54.288039 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:54.303666 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:54.370472 31034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:13:54.371255 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6f8bd03c05d84fdaaf138a6d0608e30e: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:54.403648 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:54.627346 29025 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:54.692061 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:54.726673 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:54.807632 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:54.864243 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:54.902846 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:54.926604 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:54.961758 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:55.116958 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:55.121493 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:13:55.168100 29639 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:55.233868 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:55.256007 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:55.349151 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:55.382530 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:55.519234 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:55.670248 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:55.704783 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:55.717206 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:55.727180 29639 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:55.748226 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:55.807381 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:55.857713 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 14:13:55.875790 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:55.959830 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:56.003816 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:56.018527 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:56.076048 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:56.116776 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:13:56.123785 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:56.132196 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:13:56.151191 29639 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:56.338150 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:56.416438 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:56.421787 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:56.483215 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:56.519444 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:56.544683 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:13:56.568710 29025 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:56.625936 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:56.717018 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:56.828873 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:56.859431 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:56.876044 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:56.982206 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:56.982741 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:56.991806 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:13:57.018899 29025 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:57.220726 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:57.301339 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:57.339372 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:13:57.347271 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:57.411175 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:57.455197 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:57.472636 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:57.546700 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:57.624632 30581 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:57.740428 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:57.778702 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:57.779417 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:57.787945 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:57.865816 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:57.933543 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:57.968596 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:57.969918 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:58.142225 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:58.142800 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:13:58.237977 30581 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:58.243958 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:58.314196 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:58.361940 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:58.401865 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:58.435403 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:58.475402 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:58.575232 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:58.591146 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:58.787967 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:13:58.797390 30581 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:58.813759 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:58.845969 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:58.853577 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:58.936618 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:58.991933 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:59.105137 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:59.115259 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:59.140290 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:59.253405 29025 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:59.258394 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:59.319520 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:59.327059 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:59.336441 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:59.489579 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:59.515110 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:59.534257 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:13:59.628551 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:59.809135 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:13:59.833477 29025 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:13:59.908749 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:13:59.919535 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:59.934937 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:00.016819 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:00.088449 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:14:00.225069 29025 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:00.246039 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:00.325167 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:00.349125 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:00.413667 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:00.418321 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:00.524653 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:00.644452 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:00.654074 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:00.729645 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:14:00.804350 31663 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:00.823181 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:00.830579 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:00.836912 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:00.925904 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:14:00.926440 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:01.060417 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:01.113095 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:01.138940 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:14:01.139484 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:01.151321 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:01.261884 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:01.341357 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:01.368965 32019 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:01.430112 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:01.431946 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:01.466486 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:01.500586 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:01.552882 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:01.722018 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:14:01.754663 32019 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:01.791232 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:01.838316 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:01.868010 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:01.926131 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:01.987049 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:02.026053 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:02.026887 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:14:02.098027 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:02.126575 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:14:02.176920 31663 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:02.362109 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:02.387770 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:14:02.388466 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:02.525697 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:02.580600 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:02.655463 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:14:02.687104 32327 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:02.710817 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:02.711727 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:14:02.838835 30587 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:02.838989 30587 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:14:02.839478 30587 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:14:02.839929 473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:14:02.840093 473 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:14:02.840641 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:02.840729 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:02.840780 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:14:02.840934 30587 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:02.925906 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:02.977885 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:03.014976 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:03.039494 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:03.166812 32327 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:03.202689 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:14:03.203305 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:03.290104 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:03.333307 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:03.340682 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:03.376304 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:03.387684 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:03.467180 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:03.496063 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:03.551687 32327 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:03.604425 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:03.676601 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:03.729431 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:03.866514 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:03.870071 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:03.954306 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:03.987710 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:04.016430 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:14:04.035230 32327 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:04.085690 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:04.131245 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:04.200254 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:04.249054 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:04.252012 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:04.307547 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:04.317049 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:04.368712 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:04.423075 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:14:04.451268 32327 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:04.524731 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:04.559706 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:04.842303 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:04.861580 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:04.894923 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:04.925948 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:04.994091 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:04.996488 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:14:05.052273 32327 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:05.341339 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:05.442662 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:05.488325 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:05.489569 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:05.544647 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:05.589075 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:14:05.631695 32327 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:05.768174 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:05.854605 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:05.881603 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:05.888293 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:05.894937 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:05.931389 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:05.957237 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 14:14:05.957831 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:05.970104 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:05.977216 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:06.063222 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:06.148100 870 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:06.165573 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240501 14:14:06.166275 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:06.372339 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:06.398360 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:06.410466 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:06.492014 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:06.506160 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:06.618131 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:14:06.632098 32327 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:06.638414 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:06.794828 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:06.821357 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:06.825080 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:06.836400 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:06.919858 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:06.938364 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:07.010097 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:07.074010 32327 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:14:07.155941 911 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:07.156095 911 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:14:07.156627 911 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:14:07.157130 473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
W20240501 14:14:07.157199 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240501 14:14:07.157294 473 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:14:07.157874 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:07.157961 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:07.158005 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:14:07.158138 911 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:07.200942 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:07.207342 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:07.356268 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:07.373361 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:07.403935 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:07.462966 1122 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:07.488320 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:07.621580 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:07.763784 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 14:14:07.819804 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:07.823243 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:07.857842 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:07.903975 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:08.015126 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:08.024116 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:08.072304 1347 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:08.095813 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:08.148190 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:08.194196 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:08.259728 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:08.267127 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:14:08.267799 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:08.280777 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:08.393606 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:08.459570 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:08.573974 1464 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:08.592737 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:08.661972 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:08.680090 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:08.762850 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:08.824321 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:08.881606 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:08.903419 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:14:09.035804 1464 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:09.043402 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:09.124536 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:09.124627 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:09.199476 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:09.207860 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:09.248562 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:09.345166 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:09.361975 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:09.529209 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:09.574929 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:09.585991 1347 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:09.663188 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:09.686115 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:09.731745 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:09.753599 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:10.035684 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:10.082259 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:14:10.085036 1347 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:10.201447 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:10.201985 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:10.251714 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:10.283763 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:10.327615 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:10.477738 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:14:10.558735 1347 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:10.588361 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:10.630672 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:10.659646 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:10.679077 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:10.716938 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:10.736315 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:10.756362 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:10.837101 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:11.017846 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:11.078437 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:14:11.079028 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:14:11.095487 2053 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:11.107540 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:11.114189 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:11.157044 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:11.190232 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:11.201740 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:14:11.225991 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:11.356451 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:11.431725 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:14:11.509649 1347 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:11.513674 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:11.571362 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:11.602365 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:11.745699 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:11.792400 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:11.826779 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:11.882043 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:11.885392 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:11.892721 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:12.014565 2258 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:12.033356 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:12.058087 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:12.125658 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:12.180622 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:14:12.373739 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:12.458647 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:12.468142 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:12.473315 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:14:12.624028 2409 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:12.722357 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:12.819561 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:12.832787 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:12.859766 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:14:12.936866 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:13.046343 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:13.074309 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:13.092871 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:13.126590 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:13.137540 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:14:13.166973 2409 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:13.184943 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:13.200106 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:13.239504 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:13.360353 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:13.372165 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:13.372903 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:13.493075 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:13.531656 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:13.627419 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:13.633152 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:13.763366 2662 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:13.785388 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:13.823493 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:13.899281 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:14.007454 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:14.013692 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:14.120240 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:14.189210 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:14.244097 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:14:14.251523 2409 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:14.289198 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:14.315762 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:14.324880 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:14.481428 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:14.489485 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:14.523581 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:14.625532 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:14.667424 2662 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:14.675731 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:14.822182 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:14.845057 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:14.950583 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:15.037500 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:15.211055 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:15.218344 2662 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:15.244930 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:15.250149 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:15.316699 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:15.360431 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:15.522508 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:15.528213 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:15.646292 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:15.659489 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:14:15.740109 3128 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:15.777283 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:15.781908 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:15.795670 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:15.824868 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:14:15.859062 2461 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:15.859185 2461 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:14:15.859673 2461 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:14:15.860029 473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:14:15.860178 473 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:14:15.860790 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:15.860941 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:15.860985 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:14:15.861129 2461 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:15.918656 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:15.929736 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:15.984858 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:16.090123 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:14:16.206991 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:14:16.215240 2662 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:16.217980 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:16.266610 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:14:16.298707 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:16.412279 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:16.437820 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:16.452412 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:16.469348 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:16.501456 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:16.557219 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:16.568323 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:16.620045 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:16.690596 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:16.706144 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:16.737972 3128 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:16.751497 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:16.779997 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:16.903301 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:16.909907 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:17.040244 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:17.059666 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:17.186375 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:14:17.230326 2662 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:17.316592 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:17.318092 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:17.404533 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:17.435441 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:17.486299 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:17.649436 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:14:17.676299 3128 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:17.693135 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:17.726377 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:17.900439 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:17.908349 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:17.910038 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:17.950991 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:17.951169 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:14:17.979568 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:18.067632 3128 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:18.132648 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:18.204149 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:18.235503 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:18.326525 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:18.413455 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:18.451611 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:18.479648 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:14:18.517012 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:18.568998 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:14:18.606484 3809 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:18.619763 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:18.645072 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:18.894976 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:18.917765 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:18.920400 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:18.931823 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:19.034643 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:19.052739 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:19.065388 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:14:19.083479 3809 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:19.113615 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:19.116291 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:19.219504 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:19.351173 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:19.357005 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:19.565609 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:19.573709 3809 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:19.626252 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:19.644150 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:19.687577 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:19.711954 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:19.916263 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:19.924854 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:14:19.968253 3128 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:20.047628 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:20.074631 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:20.133644 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:20.170325 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:20.259255 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:20.398834 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:20.434906 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:20.438692 3809 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:20.449376 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:20.457722 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:20.550355 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:20.596596 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:20.598781 3343 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:20.598909 3343 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:14:20.599514 3343 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:14:20.599905 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "b934fd09590a40ada094b9715d990a00" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:14:20.600072 31580 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b934fd09590a40ada094b9715d990a00 in term 2.
W20240501 14:14:20.600663 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:20.600785 411 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:20.600862 411 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:14:20.601009 3343 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:20.674422 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:20.689007 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:20.716332 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:20.897714 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:20.920898 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:20.939810 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:20.966466 4352 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:21.076249 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:21.129746 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:14:21.130403 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:21.149188 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:21.288028 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:21.301389 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:14:21.326879 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:21.410108 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:21.471642 4471 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:21.472084 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:21.489619 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:21.521070 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:21.559825 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:21.562567 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:21.646656 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:21.772094 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:21.774312 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:21.847967 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:22.037657 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:22.069083 4603 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:22.169234 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:22.236586 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:22.316054 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:14:22.332355 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:22.353224 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:22.356804 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:22.374464 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:22.392885 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:22.474107 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:22.666507 4757 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:22.689853 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:22.878459 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:22.878823 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:22.897205 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:22.962894 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:22.972219 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:22.972596 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 14:14:23.000449 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:23.003307 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:23.078094 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:23.095403 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:23.259886 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:14:23.266961 4757 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:23.344635 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:23.381223 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:23.384899 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:23.561767 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:23.585660 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:14:23.621796 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:23.680289 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:14:23.700886 4352 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:23.740819 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:23.764421 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:23.831811 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:23.926331 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:23.973737 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:23.986968 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:24.009091 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:24.016922 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:24.058111 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:24.078972 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:24.140446 4352 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:24.198792 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:24.308264 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:24.437012 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:24.572970 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:24.612493 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:24.715984 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:24.718708 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:24.726414 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:14:24.761906 5228 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:24.774817 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:24.948683 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:24.971544 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:25.003054 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:25.073347 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:25.170905 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:25.285128 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:25.323837 5228 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:25.387720 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:25.408538 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:25.450434 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:25.450990 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:25.543087 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:25.604511 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:25.632063 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:25.653628 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:25.709168 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:25.806198 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:14:25.873607 5479 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:25.908459 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:26.049463 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:26.093654 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:26.151371 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 14:14:26.152040 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:26.225929 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:26.280522 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:26.290556 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:26.337833 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:14:26.338480 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:14:26.354390 5479 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:26.417481 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:26.465904 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:26.494300 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:26.509004 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:26.523602 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:26.663735 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:26.737437 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:26.779719 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:26.871984 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:26.889485 5721 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:26.997556 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:27.080341 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:27.148777 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:27.247434 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:27.261359 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:27.273612 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:27.291623 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:14:27.296495 5721 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:27.469890 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:27.482162 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:14:27.482843 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:27.555258 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:27.663573 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:27.768584 5721 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:27.778741 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:27.883867 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:27.886327 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:27.979516 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:28.010922 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:28.059338 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:28.078168 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:28.079967 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:28.127405 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:14:28.155447 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:28.236260 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:28.361035 6045 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:28.364078 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:28.414489 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:28.428628 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:28.445556 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:28.575407 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:28.758082 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:28.779541 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:28.798480 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:14:28.807602 6045 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:28.814931 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:28.818730 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:14:28.819348 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:28.924649 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:28.983135 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:28.985054 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:29.105589 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:29.130314 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:29.339735 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:29.340039 6280 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:29.381381 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:29.389290 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:29.404913 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:29.429147 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:29.433164 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:29.547062 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:29.678046 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:14:29.812815 6045 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:29.834709 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:29.868494 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:29.895855 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:29.969611 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:30.004454 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:30.041615 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:30.313359 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:30.313575 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:14:30.335665 6488 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:30.371100 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:30.405941 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:30.413763 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:30.432130 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:30.441514 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:30.445858 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:30.782315 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:30.829383 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:30.840260 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:30.848809 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:30.852564 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:30.886046 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:30.941504 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:14:30.947441 6045 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:31.146270 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:31.177961 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 14:14:31.236356 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:31.244783 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:31.298753 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:31.305800 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:31.355180 6637 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:31.429456 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:14:31.465371 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:31.490576 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:31.569591 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:31.595556 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:31.613013 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:31.660589 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:31.709600 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:31.738943 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:31.815465 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:31.896991 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:31.946669 6891 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:31.998129 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:32.042969 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:32.084986 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:32.160750 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:32.193933 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:32.312958 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:32.331219 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:32.485879 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:14:32.486490 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:32.569074 7023 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:32.599144 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:32.624063 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:32.655035 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:32.701578 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:32.721349 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:32.883352 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:32.886440 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:33.009402 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:33.012718 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:33.032969 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:33.061583 6891 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:33.070477 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:33.129568 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:14:33.149112 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:33.204126 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:33.209187 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:33.454135 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:33.466820 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:33.622685 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:33.665280 7278 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:33.683203 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:33.692689 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:33.788375 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:33.875639 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:33.973326 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:14:33.999091 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:34.019487 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:14:34.036675 7329 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:34.036855 7329 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:14:34.037446 7329 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:14:34.037830 473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:14:34.037974 473 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:14:34.038516 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:34.038600 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:34.038633 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:14:34.038760 7329 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:34.041639 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:34.074856 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:34.129362 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:14:34.138967 7278 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:34.147617 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:34.185045 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:34.236362 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:34.267704 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:34.351218 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:34.490134 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:34.496258 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:34.523680 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:34.576570 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:34.606971 7278 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:34.640386 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:34.706656 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:34.743185 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:35.094390 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:35.104640 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:35.114153 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:35.160328 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:35.190007 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:14:35.210726 7278 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:35.214711 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:35.426151 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:35.436228 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:35.501040 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:35.573395 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:14:35.591938 7634 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:35.623127 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:35.627516 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:35.684170 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:35.780054 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:35.819067 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:35.995796 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:36.022375 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:36.079319 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:36.099747 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:36.135309 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:36.146860 7857 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:36.214948 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:36.374583 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:14:36.484692 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:14:36.485618 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:36.528447 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:36.537745 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:36.551664 7857 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:36.591609 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:36.598918 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:36.643805 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:36.796002 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:36.805737 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:36.829124 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:36.948253 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:36.964615 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:36.966068 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:36.983225 7857 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:36.985442 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:37.057761 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:37.081091 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:37.097110 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:37.204676 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:37.232263 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:37.441409 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:37.450963 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:37.455899 7857 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:37.499125 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:37.512756 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:37.564904 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:14:37.609953 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:37.812269 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:37.836126 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:37.842036 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:37.874408 7857 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:37.892685 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:37.994689 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:38.021317 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:38.023480 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:38.168897 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:38.253263 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:38.277662 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:14:38.343819 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:38.365891 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:38.410648 8364 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:38.506211 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:38.528465 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:38.549551 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:38.613169 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:38.785362 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:38.800081 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:38.893122 8364 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:38.994383 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:39.025439 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:39.065622 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:39.107861 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:39.146060 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:39.174156 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:39.195497 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:39.353224 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:39.355166 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:39.402118 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:39.408712 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:39.416251 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:14:39.467669 8364 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:39.538836 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:39.548646 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:39.591663 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:39.641877 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:39.778640 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:39.826819 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:39.849581 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:39.930680 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:39.974017 8364 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:40.025539 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:40.159014 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:40.160331 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:40.213603 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:40.241184 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:40.315452 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:40.372813 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:40.419894 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:40.459416 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:14:40.557677 8851 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:40.666271 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:40.777014 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:40.810422 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:40.835875 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:40.838917 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:40.937217 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:40.978622 8364 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:41.062583 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:41.076115 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:41.148792 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:41.255009 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:41.327535 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:41.384748 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:14:41.385344 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:41.420532 8851 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:41.452490 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:41.542032 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:14:41.554409 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:41.562919 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:41.600906 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:41.662096 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:41.710549 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:41.813213 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:41.854831 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:41.906695 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:41.908166 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:41.923416 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:14:41.954851 9174 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:41.997506 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:42.044562 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:42.207394 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:42.209583 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:42.230518 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:42.317575 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:42.377585 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:42.400882 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:42.443415 8892 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:42.443545 8892 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:14:42.443959 8892 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:14:42.444291 472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c082ccc777ba4f9a937931f707645e54" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b934fd09590a40ada094b9715d990a00" is_pre_election: true
I20240501 14:14:42.444434 472 raft_consensus.cc:2454] T c082ccc777ba4f9a937931f707645e54 P b934fd09590a40ada094b9715d990a00 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:14:42.445060 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:42.445223 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:42.445269 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:14:42.445407 8892 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:42.455508 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:42.544449 9174 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:42.569213 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:42.623781 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:14:42.670074 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:42.710431 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:42.731638 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:42.800503 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:42.877506 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:43.019181 9174 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:43.066295 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:43.139102 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:43.321846 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:43.324122 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:43.328425 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:43.413916 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:43.431066 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:14:43.439486 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:43.452679 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:43.474438 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:14:43.539732 9174 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:43.603940 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:43.737977 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:43.851060 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:43.885246 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:43.918386 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:43.931758 8851 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:44.039726 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:44.053857 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:44.081274 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:14:44.127204 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:44.128604 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:44.140866 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:44.219451 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:44.302243 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:44.312876 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:44.335156 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:44.420670 9174 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:44.433354 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:44.472992 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:44.602584 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:44.603955 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:44.637174 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:44.638058 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:14:44.765498 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:44.766010 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:44.962725 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:14:45.027297 9861 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:45.070396 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:45.079293 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:45.136086 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:45.144042 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:45.221295 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:45.299621 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:45.416916 9174 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:45.446516 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:14:45.484952 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:45.550897 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:45.637326 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:45.684008 31518 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:45.800072 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:14:45.815150 9861 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:45.887403 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:45.959965 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:46.004261 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:14:46.067804 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:46.081619 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:46.136199 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:46.261296 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:46.270381 9861 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:46.402364 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 14:14:46.509837 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:46.625852 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:46.659636 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 14:14:46.659775 9861 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:46.660324 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:46.674707 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:46.723964 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:14:46.724349 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:46.750290 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:46.815204 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:46.836125 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:46.859112 411 consensus_peers.cc:485] T 96f5e4e25a674a99a6762757f0ab1545 P b934fd09590a40ada094b9715d990a00 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:46.875377 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:46.919221 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:14:47.094801 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:47.123286 9174 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:47.186071 31256 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 14:14:47.206748 9861 consensus_queue.cc:579] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Leader has been unable to successfully communicate with peer a63e0dd1ba2e4a8db8ace9718a700c32 for more than 300 seconds (300.128s)
W20240501 14:14:47.207918 31125 consensus_peers.cc:462] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:47.212409 31188 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:14:47.213778 737 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240501 14:14:47.214238 9861 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240501 14:14:47.214515 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:47.215955 10368 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:14:47.216356 737 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } }
I20240501 14:14:47.217824 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1006490ec2841b2a3365295caaf19e1 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 14:14:47.217869 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:47.218931 31041 catalog_manager.cc:5484] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 15 to 16, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:47.224900 31188 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } }
W20240501 14:14:47.226708 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:47.226862 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): Couldn't send request to peer 6a6fcc099d1340c383e5a429643d5b7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1006490ec2841b2a3365295caaf19e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:47.228019 737 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20240501 14:14:47.228360 9861 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20240501 14:14:47.229571 31034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 3 similar messages]
I20240501 14:14:47.229857 9174 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } } }
I20240501 14:14:47.230266 737 raft_consensus.cc:2932] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } } }
W20240501 14:14:47.230418 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b1006490ec2841b2a3365295caaf19e1: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:47.231370 31031 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b1006490ec2841b2a3365295caaf19e1 with cas_config_opid_index 16: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:47.232698 31046 catalog_manager.cc:5484] T b1006490ec2841b2a3365295caaf19e1 P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 16 to 17, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } } }
I20240501 14:14:47.259244 9709 consensus_queue.cc:579] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Leader has been unable to successfully communicate with peer a63e0dd1ba2e4a8db8ace9718a700c32 for more than 300 seconds (300.362s)
W20240501 14:14:47.260593 31518 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:47.266294 31580 consensus_queue.cc:237] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
W20240501 14:14:47.266523 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:14:47.268059 472 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:14:47.268815 31450 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:14:47.268800 10390 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:14:47.269668 10390 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:14:47.271196 10389 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:14:47.272154 472 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:14:47.274170 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:14:47.274493 31450 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:14:47.274940 31041 catalog_manager.cc:5484] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 7 to 8, NON_VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:14:47.278892 31580 consensus_queue.cc:237] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } }
I20240501 14:14:47.280320 31450 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:14:47.280689 10389 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:14:47.280961 472 raft_consensus.cc:1261] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 14:14:47.281936 31516 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b83f4cd48b4b4251a8ac4ddebdb8e4e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:47.282866 10389 consensus_queue.cc:1035] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:14:47.284300 10390 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:14:47.284669 31450 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:14:47.285373 10317 consensus_queue.cc:579] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Leader has been unable to successfully communicate with peer a63e0dd1ba2e4a8db8ace9718a700c32 for more than 300 seconds (300.152s)
I20240501 14:14:47.285323 472 raft_consensus.cc:2932] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:14:47.286934 31033 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:47.287519 31039 catalog_manager.cc:5484] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 8 to 9, NON_VOTER a0caf2974041408bb32362bfc4284eaa (127.23.191.71) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: NON_VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: true } } }
I20240501 14:14:47.313158 10378 consensus_queue.cc:579] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Leader has been unable to successfully communicate with peer a63e0dd1ba2e4a8db8ace9718a700c32 for more than 300 seconds (300.411s)
I20240501 14:14:47.315860 31188 consensus_queue.cc:237] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
W20240501 14:14:47.317546 31124 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bca09920a7904362bc546916e3f6f5ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:47.317553 737 raft_consensus.cc:1261] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 14:14:47.318217 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:47.318379 10378 consensus_queue.cc:1035] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:14:47.320168 9861 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:14:47.320617 737 raft_consensus.cc:2932] T bca09920a7904362bc546916e3f6f5ac P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:14:47.322129 31031 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bca09920a7904362bc546916e3f6f5ac with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:47.322803 31041 catalog_manager.cc:5484] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 9 to 10, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:47.346567 9613 consensus_queue.cc:579] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Leader has been unable to successfully communicate with peer a63e0dd1ba2e4a8db8ace9718a700c32 for more than 300 seconds (300.071s)
I20240501 14:14:47.352542 31450 consensus_queue.cc:237] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } }
I20240501 14:14:47.354023 472 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:14:47.354398 10456 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:14:47.354665 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265): Couldn't send request to peer 30e0b8849e3841ca9ab0de01e719ee4a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:47.354738 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:47.356052 10463 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } } }
I20240501 14:14:47.356493 472 raft_consensus.cc:2932] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } } }
I20240501 14:14:47.357743 31032 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5aa7bdac7bd1408cbf41d894d40aebd4 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:47.358877 31046 catalog_manager.cc:5484] T 5aa7bdac7bd1408cbf41d894d40aebd4 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 9 to 10, NON_VOTER 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66) added. New cstate: current_term: 1 leader_uuid: "6a6fcc099d1340c383e5a429643d5b7c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } } }
W20240501 14:14:47.524673 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:14:47.531096 10368 consensus_queue.cc:579] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Leader has been unable to successfully communicate with peer a63e0dd1ba2e4a8db8ace9718a700c32 for more than 300 seconds (300.160s)
I20240501 14:14:47.579525 10368 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:14:47.635818 9810 consensus_queue.cc:579] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Leader has been unable to successfully communicate with peer 9438789dc61f4501ad1d4ceaeccee458 for more than 300 seconds (300.410s)
W20240501 14:14:47.637137 31256 consensus_peers.cc:462] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:47.641450 31319 consensus_queue.cc:237] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
I20240501 14:14:47.642704 737 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:14:47.642841 472 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:14:47.643087 9810 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:14:47.643235 10505 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:14:47.644878 10505 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:14:47.645370 737 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:14:47.645547 472 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:14:47.646227 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5d994081c3414f779ba2cae431cb1d97 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:14:47.647806 31041 catalog_manager.cc:5484] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa reported cstate change: config changed from index 10 to 11, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:14:47.651348 31319 consensus_queue.cc:237] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } }
I20240501 14:14:47.652561 737 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:14:47.652745 472 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240501 14:14:47.652905 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349): Couldn't send request to peer e7eada5032fb45c3a5e3196e2b3faaab. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:47.653110 10317 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:14:47.653292 9810 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:14:47.654652 10317 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:14:47.655093 472 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:14:47.656056 31034 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5d994081c3414f779ba2cae431cb1d97 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:47.657121 737 raft_consensus.cc:2932] T 5d994081c3414f779ba2cae431cb1d97 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } } }
I20240501 14:14:47.657298 31046 catalog_manager.cc:5484] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 11 to 12, NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:47.664573 10389 consensus_queue.cc:579] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Leader has been unable to successfully communicate with peer a63e0dd1ba2e4a8db8ace9718a700c32 for more than 300 seconds (300.029s)
W20240501 14:14:47.665222 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:14:47.666894 31580 consensus_queue.cc:237] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:14:47.667878 10368 consensus_queue.cc:579] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Leader has been unable to successfully communicate with peer 9438789dc61f4501ad1d4ceaeccee458 for more than 300 seconds (300.299s)
I20240501 14:14:47.668092 737 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:14:47.669014 10390 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:14:47.668802 31450 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 14:14:47.669296 31125 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:47.669763 10390 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:14:47.670550 31188 consensus_queue.cc:237] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } }
I20240501 14:14:47.671733 10389 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:14:47.672086 31450 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:14:47.672866 737 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
I20240501 14:14:47.673527 31319 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:14:47.673646 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e68be1521964ba8b864801fdf3b0df3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:14:47.674019 10378 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:14:47.674934 31039 catalog_manager.cc:5484] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c reported cstate change: config changed from index 8 to 9, VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } }
W20240501 14:14:47.675738 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:47.676689 9174 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } }
I20240501 14:14:47.677171 31319 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } }
I20240501 14:14:47.678671 31580 consensus_queue.cc:237] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:14:47.679116 31031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c0b7ede9ffaf496e9b106b5d05ed25a5 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:14:47.679762 737 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:14:47.679832 31450 raft_consensus.cc:1261] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:14:47.680079 10389 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:14:47.680233 10390 consensus_queue.cc:1035] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:14:47.680750 31517 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e68be1521964ba8b864801fdf3b0df3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:47.680727 31041 catalog_manager.cc:5484] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 10 to 11, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } }
I20240501 14:14:47.682149 10389 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:14:47.682736 737 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P a0caf2974041408bb32362bfc4284eaa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:14:47.683557 31450 raft_consensus.cc:2932] T 2e68be1521964ba8b864801fdf3b0df3 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:14:47.684115 31033 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2e68be1521964ba8b864801fdf3b0df3 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:47.684734 31041 catalog_manager.cc:5484] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 9 to 10, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:47.686156 31188 consensus_queue.cc:237] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:14:47.687377 31319 raft_consensus.cc:1261] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240501 14:14:47.687686 31124 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0b7ede9ffaf496e9b106b5d05ed25a5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:47.687990 9174 consensus_queue.cc:1035] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 14:14:47.688370 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:47.689721 9861 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:14:47.690317 31319 raft_consensus.cc:2932] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:14:47.691226 31031 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c0b7ede9ffaf496e9b106b5d05ed25a5 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:47.692113 31046 catalog_manager.cc:5484] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 11 to 12, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:14:47.707346 411 consensus_peers.cc:485] T 5b2e30cb9de14fe19273b3cef801cb67 P b934fd09590a40ada094b9715d990a00 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:14:47.761591 10545 ts_tablet_manager.cc:902] T bca09920a7904362bc546916e3f6f5ac P b934fd09590a40ada094b9715d990a00: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
F20240501 14:14:47.761828 10545 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-3c8200d9c20e4140906c71c746badb94.queue_time_us, tablet-download-pool-3c8200d9c20e4140906c71c746badb94.queue_time_us) (tablet-download-pool-3c8200d9c20e4140906c71c746badb94.run_cpu_time_us, tablet-download-pool-3c8200d9c20e4140906c71c746badb94.run_cpu_time_us) (tablet-download-pool-3c8200d9c20e4140906c71c746badb94.run_wall_time_us, tablet-download-pool-3c8200d9c20e4140906c71c746badb94.run_wall_time_us) (tablet-download-pool-4603d1b1b1d441b1b0be0464e2f2c9e2.queue_time_us, tablet-download-pool-4603d1b1b1d441b1b0be0464e2f2c9e2.queue_time_us) (tablet-download-pool-4603d1b1b1d441b1b0be0464e2f2c9e2.run_cpu_time_us, tablet-download-pool-4603d1b1b1d441b1b0be0464e2f2c9e2.run_cpu_time_us) (tablet-download-pool-4603d1b1b1d441b1b0be0464e2f2c9e2.run_wall_time_us, tablet-download-pool-4603d1b1b1d441b1b0be0464e2f2c9e2.run_wall_time_us) (tablet-download-pool-5aa7bdac7bd1408cbf41d894d40aebd4.queue_time_us, tablet-download-pool-5aa7bdac7bd1408cbf41d894d40aebd4.queue_time_us) (tablet-download-pool-5aa7bdac7bd1408cbf41d894d40aebd4.run_cpu_time_us, tablet-download-pool-5aa7bdac7bd1408cbf41d894d40aebd4.run_cpu_time_us) (tablet-download-pool-5aa7bdac7bd1408cbf41d894d40aebd4.run_wall_time_us, tablet-download-pool-5aa7bdac7bd1408cbf41d894d40aebd4.run_wall_time_us) (tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.queue_time_us, tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.queue_time_us) (tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.run_cpu_time_us, tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.run_cpu_time_us) (tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.run_wall_time_us, tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.run_wall_time_us) (tablet-download-pool-5d994081c3414f779ba2cae431cb1d97.queue_time_us, tablet-download-pool-5d994081c3414f779ba2cae431cb1d97.queue_time_us) (tablet-download-pool-5d994081c3414f779ba2cae431cb1d97.run_cpu_time_us, tablet-download-pool-5d994081c3414f779ba2cae431cb1d97.run_cpu_time_us) (tablet-download-pool-5d994081c3414f779ba2cae431cb1d97.run_wall_time_us, tablet-download-pool-5d994081c3414f779ba2cae431cb1d97.run_wall_time_us) (tablet-download-pool-72ad9c714b2b4fff94063d2838835173.queue_time_us, tablet-download-pool-72ad9c714b2b4fff94063d2838835173.queue_time_us) (tablet-download-pool-72ad9c714b2b4fff94063d2838835173.run_cpu_time_us, tablet-download-pool-72ad9c714b2b4fff94063d2838835173.run_cpu_time_us) (tablet-download-pool-72ad9c714b2b4fff94063d2838835173.run_wall_time_us, tablet-download-pool-72ad9c714b2b4fff94063d2838835173.run_wall_time_us) (tablet-download-pool-8ffdb76b2c5149a18d45bee9581e605a.queue_time_us, tablet-download-pool-8ffdb76b2c5149a18d45bee9581e605a.queue_time_us) (tablet-download-pool-8ffdb76b2c5149a18d45bee9581e605a.run_cpu_time_us, tablet-download-pool-8ffdb76b2c5149a18d45bee9581e605a.run_cpu_time_us) (tablet-download-pool-8ffdb76b2c5149a18d45bee9581e605a.run_wall_time_us, tablet-download-pool-8ffdb76b2c5149a18d45bee9581e605a.run_wall_time_us) (tablet-download-pool-96f5e4e25a674a99a6762757f0ab1545.queue_time_us, tablet-download-pool-96f5e4e25a674a99a6762757f0ab1545.queue_time_us) (tablet-download-pool-96f5e4e25a674a99a6762757f0ab1545.run_cpu_time_us, tablet-download-pool-96f5e4e25a674a99a6762757f0ab1545.run_cpu_time_us) (tablet-download-pool-96f5e4e25a674a99a6762757f0ab1545.run_wall_time_us, tablet-download-pool-96f5e4e25a674a99a6762757f0ab1545.run_wall_time_us) (tablet-download-pool-9d2c173b2c32494eb16e46a51ca36e31.queue_time_us, tablet-download-pool-9d2c173b2c32494eb16e46a51ca36e31.queue_time_us) (tablet-download-pool-9d2c173b2c32494eb16e46a51ca36e31.run_cpu_time_us, tablet-download-pool-9d2c173b2c32494eb16e46a51ca36e31.run_cpu_time_us) (tablet-download-pool-9d2c173b2c32494eb16e46a51ca36e31.run_wall_time_us, tablet-download-pool-9d2c173b2c32494eb16e46a51ca36e31.run_wall_time_us) (tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.queue_time_us, tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.queue_time_us) (tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.run_cpu_time_us, tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.run_cpu_time_us) (tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.run_wall_time_us, tablet-download-pool-a7810938b8fb406e9c6ac438f54afd03.run_wall_time_us) (tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.queue_time_us, tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.queue_time_us) (tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_cpu_time_us, tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_cpu_time_us) (tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_wall_time_us, tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_wall_time_us) (tablet-download-pool-b83f4cd48b4b4251a8ac4ddebdb8e4e1.queue_time_us, tablet-download-pool-b83f4cd48b4b4251a8ac4ddebdb8e4e1.queue_time_us) (tablet-download-pool-b83f4cd48b4b4251a8ac4ddebdb8e4e1.run_cpu_time_us, tablet-download-pool-b83f4cd48b4b4251a8ac4ddebdb8e4e1.run_cpu_time_us) (tablet-download-pool-b83f4cd48b4b4251a8ac4ddebdb8e4e1.run_wall_time_us, tablet-download-pool-b83f4cd48b4b4251a8ac4ddebdb8e4e1.run_wall_time_us) (tablet-download-pool-bca09920a7904362bc546916e3f6f5ac.queue_time_us, tablet-download-pool-bca09920a7904362bc546916e3f6f5ac.queue_time_us) (tablet-download-pool-c082ccc777ba4f9a937931f707645e54.queue_time_us, tablet-download-pool-c082ccc777ba4f9a937931f707645e54.queue_time_us) (tablet-download-pool-c082ccc777ba4f9a937931f707645e54.run_cpu_time_us, tablet-download-pool-c082ccc777ba4f9a937931f707645e54.run_cpu_time_us) (tablet-download-pool-c082ccc777ba4f9a937931f707645e54.run_wall_time_us, tablet-download-pool-c082ccc777ba4f9a937931f707645e54.run_wall_time_us) (tablet-download-pool-f26d16499ab54a3e9059f6e4ff041177.queue_time_us, tablet-download-pool-f26d16499ab54a3e9059f6e4ff041177.queue_time_us) (tablet-download-pool-f26d16499ab54a3e9059f6e4ff041177.run_cpu_time_us, tablet-download-pool-f26d16499ab54a3e9059f6e4ff041177.run_cpu_time_us) (tablet-download-pool-f26d16499ab54a3e9059f6e4ff041177.run_wall_time_us, tablet-download-pool-f26d16499ab54a3e9059f6e4ff041177.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f9b5d7d2dcd google::LogMessage::Fail() at ??:0
I20240501 14:14:47.765259 10547 ts_tablet_manager.cc:902] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c: Initiating tablet copy from peer 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841)
I20240501 14:14:47.765780 10547 tablet_copy_client.cc:308] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.65:32841
@ 0x7f9b5d7d6b93 google::LogMessage::SendToLog() at ??:0
I20240501 14:14:47.766270 31209 tablet_copy_service.cc:140] P 0e4622d80cc94f12b492bdd7597a61e1: Received BeginTabletCopySession request for tablet b1006490ec2841b2a3365295caaf19e1 from peer 6a6fcc099d1340c383e5a429643d5b7c ({username='slave'} at 127.23.191.67:35893)
I20240501 14:14:47.766386 31209 tablet_copy_service.cc:161] P 0e4622d80cc94f12b492bdd7597a61e1: Beginning new tablet copy session on tablet b1006490ec2841b2a3365295caaf19e1 from peer 6a6fcc099d1340c383e5a429643d5b7c at {username='slave'} at 127.23.191.67:35893: session id = 6a6fcc099d1340c383e5a429643d5b7c-b1006490ec2841b2a3365295caaf19e1
@ 0x7f9b5d7d27cc google::LogMessage::Flush() at ??:0
I20240501 14:14:47.767928 31209 tablet_copy_source_session.cc:205] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:14:47.769181 10547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1006490ec2841b2a3365295caaf19e1. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f9b5d7d3f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9b5ec3c044 kudu::TraceMetrics::InternName() at ??:0
I20240501 14:14:47.771726 10547 tablet_copy_client.cc:783] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Starting download of 5 data blocks...
I20240501 14:14:47.772055 10505 consensus_queue.cc:579] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Leader has been unable to successfully communicate with peer 9438789dc61f4501ad1d4ceaeccee458 for more than 300 seconds (300.240s)
@ 0x7f9b5ec20f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240501 14:14:47.775722 31319 consensus_queue.cc:237] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } }
@ 0x7f9b5ec1eefc kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 14:14:47.777216 472 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:14:47.777709 10317 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:14:47.777930 31580 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:14:47.778981 10317 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
@ 0x7f9b617d177a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 14:14:47.780251 10547 tablet_copy_client.cc:647] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:14:47.781515 10547 tablet_copy_client.cc:523] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f9b617daa45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 14:14:47.783145 10547 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:14:47.783025 10317 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:14:47.785315 31034 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f26d16499ab54a3e9059f6e4ff041177 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:14:47.785976 31041 catalog_manager.cc:5484] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 7 to 8, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
@ 0x7f9b6185b76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 14:14:47.788228 472 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
I20240501 14:14:47.790544 31319 consensus_queue.cc:237] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } }
I20240501 14:14:47.791394 31580 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } }
@ 0x7f9b6186f4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
W20240501 14:14:47.793540 31254 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413): Couldn't send request to peer 6a6fcc099d1340c383e5a429643d5b7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f26d16499ab54a3e9059f6e4ff041177. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:47.794450 31580 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
@ 0x7f9b6185a226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240501 14:14:47.794950 472 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:14:47.795289 9810 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:14:47.795540 9810 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:14:47.797343 10317 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } } }
@ 0x7f9b61869c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55eb318e375c std::function<>::operator()() at ??:0
I20240501 14:14:47.799441 472 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } } }
@ 0x7f9b5ec23e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 14:14:47.801160 31580 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } } }
I20240501 14:14:47.803050 31041 catalog_manager.cc:5484] T f26d16499ab54a3e9059f6e4ff041177 P b934fd09590a40ada094b9715d990a00 reported cstate change: config changed from index 8 to 9, NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) added. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } } }
@ 0x7f9b5ec246ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240501 14:14:47.804137 31034 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f26d16499ab54a3e9059f6e4ff041177 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:47.805613 10547 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
@ 0x7f9b5ec2603b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240501 14:14:47.807870 10547 tablet_bootstrap.cc:492] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap complete.
@ 0x55eb318e375c std::function<>::operator()() at ??:0
I20240501 14:14:47.808051 10547 ts_tablet_manager.cc:1367] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.006s
I20240501 14:14:47.809576 10547 raft_consensus.cc:348] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } }
I20240501 14:14:47.809695 10547 raft_consensus.cc:729] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: LEARNER
I20240501 14:14:47.809785 10547 consensus_queue.cc:260] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 1.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } }
I20240501 14:14:47.810079 10547 ts_tablet_manager.cc:1398] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:14:47.810712 31209 tablet_copy_service.cc:342] P 0e4622d80cc94f12b492bdd7597a61e1: Request end of tablet copy session 6a6fcc099d1340c383e5a429643d5b7c-b1006490ec2841b2a3365295caaf19e1 received from {username='slave'} at 127.23.191.67:35893
@ 0x7f9b5ec13807 kudu::Thread::SuperviseThread() at ??:0
I20240501 14:14:47.810792 31209 tablet_copy_service.cc:433] P 0e4622d80cc94f12b492bdd7597a61e1: ending tablet copy session 6a6fcc099d1340c383e5a429643d5b7c-b1006490ec2841b2a3365295caaf19e1 on tablet b1006490ec2841b2a3365295caaf19e1 with peer 6a6fcc099d1340c383e5a429643d5b7c
@ 0x7f9b5f4516db start_thread at ??:0
@ 0x7f9b5c98971f clone at ??:0
I20240501 14:14:47.833010 10576 ts_tablet_manager.cc:902] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: Initiating tablet copy from peer 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413)
I20240501 14:14:47.833587 10576 tablet_copy_client.cc:308] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.67:39413
I20240501 14:14:47.834053 31471 tablet_copy_service.cc:140] P 6a6fcc099d1340c383e5a429643d5b7c: Received BeginTabletCopySession request for tablet 5aa7bdac7bd1408cbf41d894d40aebd4 from peer 30e0b8849e3841ca9ab0de01e719ee4a ({username='slave'} at 127.23.191.66:53563)
I20240501 14:14:47.834157 31471 tablet_copy_service.cc:161] P 6a6fcc099d1340c383e5a429643d5b7c: Beginning new tablet copy session on tablet 5aa7bdac7bd1408cbf41d894d40aebd4 from peer 30e0b8849e3841ca9ab0de01e719ee4a at {username='slave'} at 127.23.191.66:53563: session id = 30e0b8849e3841ca9ab0de01e719ee4a-5aa7bdac7bd1408cbf41d894d40aebd4
I20240501 14:14:47.835235 31471 tablet_copy_source_session.cc:205] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:14:47.836091 10576 ts_tablet_manager.cc:1875] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:14:47.838002 10576 ts_tablet_manager.cc:1888] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240501 14:14:47.838135 10576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa7bdac7bd1408cbf41d894d40aebd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:14:47.839180 10576 tablet_copy_client.cc:783] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet copy: Starting download of 5 data blocks...
I20240501 14:14:47.844072 10576 tablet_copy_client.cc:647] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:14:47.845219 10576 tablet_copy_client.cc:523] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:14:47.846432 10576 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap starting.
I20240501 14:14:47.854877 10576 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:14:47.855114 10576 tablet_bootstrap.cc:492] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: Bootstrap complete.
I20240501 14:14:47.855291 10576 ts_tablet_manager.cc:1367] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240501 14:14:47.855736 10576 raft_consensus.cc:348] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } }
I20240501 14:14:47.855863 10576 raft_consensus.cc:729] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 30e0b8849e3841ca9ab0de01e719ee4a, State: Initialized, Role: LEARNER
I20240501 14:14:47.855942 10576 consensus_queue.cc:260] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: NON_VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: true } }
I20240501 14:14:47.856185 10576 ts_tablet_manager.cc:1398] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:14:47.856925 31471 tablet_copy_service.cc:342] P 6a6fcc099d1340c383e5a429643d5b7c: Request end of tablet copy session 30e0b8849e3841ca9ab0de01e719ee4a-5aa7bdac7bd1408cbf41d894d40aebd4 received from {username='slave'} at 127.23.191.66:53563
I20240501 14:14:47.857000 31471 tablet_copy_service.cc:433] P 6a6fcc099d1340c383e5a429643d5b7c: ending tablet copy session 30e0b8849e3841ca9ab0de01e719ee4a-5aa7bdac7bd1408cbf41d894d40aebd4 on tablet 5aa7bdac7bd1408cbf41d894d40aebd4 with peer 30e0b8849e3841ca9ab0de01e719ee4a
I20240501 14:14:47.894824 10592 ts_tablet_manager.cc:902] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P a0caf2974041408bb32362bfc4284eaa: Initiating tablet copy from peer e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349)
F20240501 14:14:47.895084 10592 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-043344ce36a74a18a0d67ff5bd473e68.queue_time_us, tablet-download-pool-043344ce36a74a18a0d67ff5bd473e68.queue_time_us) (tablet-download-pool-043344ce36a74a18a0d67ff5bd473e68.run_cpu_time_us, tablet-download-pool-043344ce36a74a18a0d67ff5bd473e68.run_cpu_time_us) (tablet-download-pool-043344ce36a74a18a0d67ff5bd473e68.run_wall_time_us, tablet-download-pool-043344ce36a74a18a0d67ff5bd473e68.run_wall_time_us) (tablet-download-pool-2e68be1521964ba8b864801fdf3b0df3.queue_time_us, tablet-download-pool-2e68be1521964ba8b864801fdf3b0df3.queue_time_us) (tablet-download-pool-2e68be1521964ba8b864801fdf3b0df3.run_cpu_time_us, tablet-download-pool-2e68be1521964ba8b864801fdf3b0df3.run_cpu_time_us) (tablet-download-pool-2e68be1521964ba8b864801fdf3b0df3.run_wall_time_us, tablet-download-pool-2e68be1521964ba8b864801fdf3b0df3.run_wall_time_us) (tablet-download-pool-4603d1b1b1d441b1b0be0464e2f2c9e2.queue_time_us, tablet-download-pool-4603d1b1b1d441b1b0be0464e2f2c9e2.queue_time_us) (tablet-download-pool-4603d1b1b1d441b1b0be0464e2f2c9e2.run_cpu_time_us, tablet-download-pool-4603d1b1b1d441b1b0be0464e2f2c9e2.run_cpu_time_us) (tablet-download-pool-4603d1b1b1d441b1b0be0464e2f2c9e2.run_wall_time_us, tablet-download-pool-4603d1b1b1d441b1b0be0464e2f2c9e2.run_wall_time_us) (tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.queue_time_us, tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.queue_time_us) (tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.run_cpu_time_us, tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.run_cpu_time_us) (tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.run_wall_time_us, tablet-download-pool-5b2e30cb9de14fe19273b3cef801cb67.run_wall_time_us) (tablet-download-pool-5d994081c3414f779ba2cae431cb1d97.queue_time_us, tablet-download-pool-5d994081c3414f779ba2cae431cb1d97.queue_time_us) (tablet-download-pool-5d994081c3414f779ba2cae431cb1d97.run_cpu_time_us, tablet-download-pool-5d994081c3414f779ba2cae431cb1d97.run_cpu_time_us) (tablet-download-pool-5d994081c3414f779ba2cae431cb1d97.run_wall_time_us, tablet-download-pool-5d994081c3414f779ba2cae431cb1d97.run_wall_time_us) (tablet-download-pool-71241540f585474a8f62da16ed2d9874.queue_time_us, tablet-download-pool-71241540f585474a8f62da16ed2d9874.queue_time_us) (tablet-download-pool-71241540f585474a8f62da16ed2d9874.run_cpu_time_us, tablet-download-pool-71241540f585474a8f62da16ed2d9874.run_cpu_time_us) (tablet-download-pool-71241540f585474a8f62da16ed2d9874.run_wall_time_us, tablet-download-pool-71241540f585474a8f62da16ed2d9874.run_wall_time_us) (tablet-download-pool-72ad9c714b2b4fff94063d2838835173.queue_time_us, tablet-download-pool-72ad9c714b2b4fff94063d2838835173.queue_time_us) (tablet-download-pool-72ad9c714b2b4fff94063d2838835173.run_cpu_time_us, tablet-download-pool-72ad9c714b2b4fff94063d2838835173.run_cpu_time_us) (tablet-download-pool-72ad9c714b2b4fff94063d2838835173.run_wall_time_us, tablet-download-pool-72ad9c714b2b4fff94063d2838835173.run_wall_time_us) (tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.queue_time_us, tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.queue_time_us) (tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.run_cpu_time_us, tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.run_cpu_time_us) (tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.run_wall_time_us, tablet-download-pool-7a542c07a3f841b0a31051a8a37083bf.run_wall_time_us) (tablet-download-pool-ad0dac76fd0c4ae6b8e0549376c98e5d.queue_time_us, tablet-download-pool-ad0dac76fd0c4ae6b8e0549376c98e5d.queue_time_us) (tablet-download-pool-ad0dac76fd0c4ae6b8e0549376c98e5d.run_cpu_time_us, tablet-download-pool-ad0dac76fd0c4ae6b8e0549376c98e5d.run_cpu_time_us) (tablet-download-pool-ad0dac76fd0c4ae6b8e0549376c98e5d.run_wall_time_us, tablet-download-pool-ad0dac76fd0c4ae6b8e0549376c98e5d.run_wall_time_us) (tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.queue_time_us, tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.queue_time_us) (tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_cpu_time_us, tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_cpu_time_us) (tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_wall_time_us, tablet-download-pool-b1006490ec2841b2a3365295caaf19e1.run_wall_time_us) (tablet-download-pool-b67bd0eb45a3486492b75ca3ad3be524.queue_time_us, tablet-download-pool-b67bd0eb45a3486492b75ca3ad3be524.queue_time_us) (tablet-download-pool-b67bd0eb45a3486492b75ca3ad3be524.run_cpu_time_us, tablet-download-pool-b67bd0eb45a3486492b75ca3ad3be524.run_cpu_time_us) (tablet-download-pool-b67bd0eb45a3486492b75ca3ad3be524.run_wall_time_us, tablet-download-pool-b67bd0eb45a3486492b75ca3ad3be524.run_wall_time_us) (tablet-download-pool-b83f4cd48b4b4251a8ac4ddebdb8e4e1.queue_time_us, tablet-download-pool-b83f4cd48b4b4251a8ac4ddebdb8e4e1.queue_time_us) (tablet-download-pool-bca09920a7904362bc546916e3f6f5ac.queue_time_us, tablet-download-pool-bca09920a7904362bc546916e3f6f5ac.queue_time_us) (tablet-download-pool-bca09920a7904362bc546916e3f6f5ac.run_cpu_time_us, tablet-download-pool-bca09920a7904362bc546916e3f6f5ac.run_cpu_time_us) (tablet-download-pool-bca09920a7904362bc546916e3f6f5ac.run_wall_time_us, tablet-download-pool-bca09920a7904362bc546916e3f6f5ac.run_wall_time_us) (tablet-download-pool-ea66fe3d0da04d18b961927a9c988a20.queue_time_us, tablet-download-pool-ea66fe3d0da04d18b961927a9c988a20.queue_time_us) (tablet-download-pool-ea66fe3d0da04d18b961927a9c988a20.run_cpu_time_us, tablet-download-pool-ea66fe3d0da04d18b961927a9c988a20.run_cpu_time_us) (tablet-download-pool-ea66fe3d0da04d18b961927a9c988a20.run_wall_time_us, tablet-download-pool-ea66fe3d0da04d18b961927a9c988a20.run_wall_time_us) (tablet-download-pool-f8cc4d3985224178aa008b4c9de5625b.queue_time_us, tablet-download-pool-f8cc4d3985224178aa008b4c9de5625b.queue_time_us) (tablet-download-pool-f8cc4d3985224178aa008b4c9de5625b.run_cpu_time_us, tablet-download-pool-f8cc4d3985224178aa008b4c9de5625b.run_cpu_time_us) (tablet-download-pool-f8cc4d3985224178aa008b4c9de5625b.run_wall_time_us, tablet-download-pool-f8cc4d3985224178aa008b4c9de5625b.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f6431cf7dcd google::LogMessage::Fail() at ??:0
@ 0x7f6431cfbb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f6431cf77cc google::LogMessage::Flush() at ??:0
I20240501 14:14:47.900487 10571 consensus_queue.cc:579] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Leader has been unable to successfully communicate with peer 9438789dc61f4501ad1d4ceaeccee458 for more than 300 seconds (300.190s)
@ 0x7f6431cf8f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6433161044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f6433145f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f6433143efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f6435cf677a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f6435cffa45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f6435d8076f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f6435d944f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f6435d7f226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f6435d8ec0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55fe6f8b275c std::function<>::operator()() at ??:0
@ 0x7f6433148e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f64331496ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f643314b03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55fe6f8b275c std::function<>::operator()() at ??:0
@ 0x7f6433138807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f64339766db start_thread at ??:0
@ 0x7f6430eae71f clone at ??:0
W20240501 14:14:48.013449 31386 connection.cc:535] client connection to 127.23.191.69:45429 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:14:48.013600 31255 connection.cc:535] client connection to 127.23.191.69:45429 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:14:48.014624 31517 connection.cc:535] client connection to 127.23.191.69:45429 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:14:48.014779 31124 connection.cc:535] client connection to 127.23.191.69:45429 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:14:48.015712 9912 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:14:48.014126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.191.69:45429 (local address 127.23.191.67:49681)
0501 14:14:48.014321 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0501 14:14:48.014334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 14:14:48.014446 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:14:48.015607 (+ 1161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126}
W20240501 14:14:48.015859 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.015913 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.016404 3183 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:14:48.015364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.191.69:45429 (local address 127.23.191.68:36537)
0501 14:14:48.015472 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0501 14:14:48.015485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 14:14:48.015570 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:14:48.016108 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":30}
W20240501 14:14:48.016554 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.017997 3554 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:14:48.014204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.191.69:45429 (local address 127.23.191.66:48871)
0501 14:14:48.014395 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0501 14:14:48.014409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 14:14:48.014505 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:14:48.017467 (+ 2962us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":102}
W20240501 14:14:48.018062 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:48.018126 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.018247 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:48.025156 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(b1006490ec2841b2a3365295caaf19e1): 4834 bytes on disk
I20240501 14:14:48.025581 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(b1006490ec2841b2a3365295caaf19e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240501 14:14:48.051225 9174 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:48.070264 31518 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 14:14:48.093770 31367 maintenance_manager.cc:419] P 30e0b8849e3841ca9ab0de01e719ee4a: Scheduling UndoDeltaBlockGCOp(5aa7bdac7bd1408cbf41d894d40aebd4): 4581 bytes on disk
I20240501 14:14:48.094228 31257 maintenance_manager.cc:643] P 30e0b8849e3841ca9ab0de01e719ee4a: UndoDeltaBlockGCOp(5aa7bdac7bd1408cbf41d894d40aebd4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 14:14:48.096478 31169 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:14:48.098004 31298 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:14:48.106218 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:48.109253 31561 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:48.110554 31429 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:48.127964 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:48.133132 9174 consensus_queue.cc:579] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Leader has been unable to successfully communicate with peer a63e0dd1ba2e4a8db8ace9718a700c32 for more than 300 seconds (300.134s)
W20240501 14:14:48.138653 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:48.142477 10671 ts_tablet_manager.cc:902] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
W20240501 14:14:48.143080 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:48.143100 10671 tablet_copy_client.cc:308] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:14:48.143730 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet 5d994081c3414f779ba2cae431cb1d97 from peer e7eada5032fb45c3a5e3196e2b3faaab ({username='slave'} at 127.23.191.68:40875)
I20240501 14:14:48.143846 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet 5d994081c3414f779ba2cae431cb1d97 from peer e7eada5032fb45c3a5e3196e2b3faaab at {username='slave'} at 127.23.191.68:40875: session id = e7eada5032fb45c3a5e3196e2b3faaab-5d994081c3414f779ba2cae431cb1d97
W20240501 14:14:48.144786 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:48.145056 31340 tablet_copy_source_session.cc:205] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:14:48.145931 10671 ts_tablet_manager.cc:1875] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:14:48.148186 10671 ts_tablet_manager.cc:1888] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240501 14:14:48.148350 10671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d994081c3414f779ba2cae431cb1d97. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:14:48.149147 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 14:14:48.149507 10671 tablet_copy_client.cc:783] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Starting download of 5 data blocks...
W20240501 14:14:48.151189 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.153275 31516 connection.cc:535] client connection to 127.23.191.71:35407 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:14:48.153331 31385 connection.cc:535] client connection to 127.23.191.71:35407 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:14:48.154016 31254 connection.cc:535] client connection to 127.23.191.71:35407 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:14:48.154183 9912 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:14:48.153815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.191.71:35407 (local address 127.23.191.67:40757)
0501 14:14:48.154055 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0501 14:14:48.154105 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":147}
W20240501 14:14:48.154477 3183 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:14:48.153852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.191.71:35407 (local address 127.23.191.68:57289)
0501 14:14:48.154346 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0501 14:14:48.154406 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":421}
W20240501 14:14:48.154646 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection reset by peer (error 104)
W20240501 14:14:48.154706 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.154748 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.154794 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.154846 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.154886 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.154942 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.154981 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.155012 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:48.162881 10671 tablet_copy_client.cc:647] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Starting download of 1 WAL segments...
I20240501 14:14:48.164474 10671 tablet_copy_client.cc:523] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:14:48.166158 10671 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap starting.
I20240501 14:14:48.174865 10671 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:14:48.175112 10671 tablet_bootstrap.cc:492] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Bootstrap complete.
I20240501 14:14:48.175287 10671 ts_tablet_manager.cc:1367] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 14:14:48.175663 10671 raft_consensus.cc:348] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } }
I20240501 14:14:48.175767 10671 raft_consensus.cc:729] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e7eada5032fb45c3a5e3196e2b3faaab, State: Initialized, Role: LEARNER
I20240501 14:14:48.175843 10671 consensus_queue.cc:260] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: NON_VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: true } }
I20240501 14:14:48.176118 10671 ts_tablet_manager.cc:1398] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:14:48.176316 918 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-tasklfWEhd/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:09:41.472707 927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:41.473052 927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:41.478263 927 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:09:42.574296 927 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:09:43.157009 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77c4973914b2412cbc293014310ba9c6: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:43.157114 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd036c60a12148d2ada73beb29aedcbf: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:43.157196 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 855c959aa35441b989d386caf8f8eded: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:43.157308 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:43.157368 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1f2efe25014e49d8b36ca467c0f40ef3: could not find any suitable replica to move from server 6a6fcc099d1340c383e5a429643d5b7c to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:43.157405 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd036c60a12148d2ada73beb29aedcbf: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:43.157465 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01dc7a764a349c9a4a9180993e4fbc3: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:43.157521 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbe6d56762d84933a7b7ddaf8c314b4f: could not find any suitable replica to move from server 6a6fcc099d1340c383e5a429643d5b7c to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:43.157560 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd036c60a12148d2ada73beb29aedcbf: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:43.157596 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 351ff5b1095247928fbb77893616e4a9: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:43.157686 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:43.157732 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77c4973914b2412cbc293014310ba9c6: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:43.157770 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1f2efe25014e49d8b36ca467c0f40ef3: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:43.157811 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 026cd477adec4547bee4fb8004a63348: could not find any suitable replica to move from server 6a6fcc099d1340c383e5a429643d5b7c to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:43.157858 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd036c60a12148d2ada73beb29aedcbf: could not find any suitable replica to move from server 6a6fcc099d1340c383e5a429643d5b7c to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:43.157887 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b01dc7a764a349c9a4a9180993e4fbc3: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:43.190562 927 rebalancer_tool.cc:1074] tablet ad0dac76fd0c4ae6b8e0549376c98e5d: 'e7eada5032fb45c3a5e3196e2b3faaab' -> 'a0caf2974041408bb32362bfc4284eaa' move scheduled
I20240501 14:09:43.246568 927 rebalancer_tool.cc:1074] tablet 9d2c173b2c32494eb16e46a51ca36e31: '30e0b8849e3841ca9ab0de01e719ee4a' -> '9438789dc61f4501ad1d4ceaeccee458' move scheduled
I20240501 14:09:43.293604 927 rebalancer_tool.cc:1074] tablet bca09920a7904362bc546916e3f6f5ac: '6a6fcc099d1340c383e5a429643d5b7c' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move scheduled
I20240501 14:09:43.349009 927 rebalancer_tool.cc:1074] tablet 5d994081c3414f779ba2cae431cb1d97: '0e4622d80cc94f12b492bdd7597a61e1' -> 'b934fd09590a40ada094b9715d990a00' move scheduled
I20240501 14:09:43.376864 927 rebalancer_tool.cc:1084] tablet 71241540f585474a8f62da16ed2d9874: 'e7eada5032fb45c3a5e3196e2b3faaab' -> 'a0caf2974041408bb32362bfc4284eaa' move ignored: Illegal state: tablet 71241540f585474a8f62da16ed2d9874: replica a0caf2974041408bb32362bfc4284eaa is already present
I20240501 14:09:43.376953 927 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:09:44.152899 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd036c60a12148d2ada73beb29aedcbf: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:44.153115 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd036c60a12148d2ada73beb29aedcbf: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:44.153151 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbe6d56762d84933a7b7ddaf8c314b4f: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:44.153193 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:44.153254 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd036c60a12148d2ada73beb29aedcbf: could not find any suitable replica to move from server 9438789dc61f4501ad1d4ceaeccee458 to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:44.153277 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 026cd477adec4547bee4fb8004a63348: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server a0caf2974041408bb32362bfc4284eaa
W20240501 14:09:44.153306 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1f2efe25014e49d8b36ca467c0f40ef3: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:44.153331 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bd1074f581c447ee86e97b1ef3b8687f: could not find any suitable replica to move from server 6a6fcc099d1340c383e5a429643d5b7c to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:44.153358 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbe6d56762d84933a7b7ddaf8c314b4f: could not find any suitable replica to move from server e7eada5032fb45c3a5e3196e2b3faaab to server 9438789dc61f4501ad1d4ceaeccee458
W20240501 14:09:44.153383 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:44.153407 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 855c959aa35441b989d386caf8f8eded: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server a0caf2974041408bb32362bfc4284eaa
I20240501 14:09:44.179296 927 rebalancer_tool.cc:1074] tablet 5b2e30cb9de14fe19273b3cef801cb67: '0e4622d80cc94f12b492bdd7597a61e1' -> 'a0caf2974041408bb32362bfc4284eaa' move scheduled
I20240501 14:09:44.211866 927 rebalancer_tool.cc:1074] tablet b1006490ec2841b2a3365295caaf19e1: 'e7eada5032fb45c3a5e3196e2b3faaab' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move scheduled
I20240501 14:09:44.478487 927 rebalancer_tool.cc:1074] tablet c082ccc777ba4f9a937931f707645e54: '6a6fcc099d1340c383e5a429643d5b7c' -> 'b934fd09590a40ada094b9715d990a00' move scheduled
I20240501 14:09:44.844552 927 rebalancer_tool.cc:1074] tablet 96f5e4e25a674a99a6762757f0ab1545: '0e4622d80cc94f12b492bdd7597a61e1' -> '9438789dc61f4501ad1d4ceaeccee458' move scheduled
I20240501 14:09:44.927211 927 rebalancer_tool.cc:1084] tablet 6f8bd03c05d84fdaaf138a6d0608e30e: '30e0b8849e3841ca9ab0de01e719ee4a' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move ignored: Illegal state: tablet 6f8bd03c05d84fdaaf138a6d0608e30e: replica a63e0dd1ba2e4a8db8ace9718a700c32 is already present
I20240501 14:09:44.927279 927 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:09:45.886093 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:09:45.886232 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbe6d56762d84933a7b7ddaf8c314b4f: could not find any suitable replica to move from server b934fd09590a40ada094b9715d990a00 to server 0e4622d80cc94f12b492bdd7597a61e1
W20240501 14:09:45.886325 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e134e011714149fd80d2248e61c82357: could not find any suitable replica to move from server 0e4622d80cc94f12b492bdd7597a61e1 to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:45.886377 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1f2efe25014e49d8b36ca467c0f40ef3: could not find any suitable replica to move from server 6a6fcc099d1340c383e5a429643d5b7c to server b934fd09590a40ada094b9715d990a00
W20240501 14:09:45.886430 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bd1074f581c447ee86e97b1ef3b8687f: could not find any suitable replica to move from server 30e0b8849e3841ca9ab0de01e719ee4a to server a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:09:45.973066 927 rebalancer_tool.cc:1084] tablet f8cc4d3985224178aa008b4c9de5625b: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '' move ignored: Not found: Peer with uuid e7eada5032fb45c3a5e3196e2b3faaab not found in consensus config
I20240501 14:09:45.973616 927 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:09:46.433306 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbe6d56762d84933a7b7ddaf8c314b4f: could not find any suitable replica to move from server a63e0dd1ba2e4a8db8ace9718a700c32 to server 0e4622d80cc94f12b492bdd7597a61e1
I20240501 14:09:46.522728 927 rebalancer_tool.cc:1074] tablet 7a542c07a3f841b0a31051a8a37083bf: '30e0b8849e3841ca9ab0de01e719ee4a' -> '9438789dc61f4501ad1d4ceaeccee458' move scheduled
I20240501 14:09:46.575065 927 rebalancer_tool.cc:1074] tablet ea66fe3d0da04d18b961927a9c988a20: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '0e4622d80cc94f12b492bdd7597a61e1' move scheduled
I20240501 14:09:46.636124 927 rebalancer_tool.cc:1074] tablet 72ad9c714b2b4fff94063d2838835173: '6a6fcc099d1340c383e5a429643d5b7c' -> '30e0b8849e3841ca9ab0de01e719ee4a' move scheduled
I20240501 14:09:46.798911 927 rebalancer_tool.cc:1074] tablet 8ffdb76b2c5149a18d45bee9581e605a: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '9438789dc61f4501ad1d4ceaeccee458' move scheduled
I20240501 14:09:46.891942 927 rebalancer_tool.cc:1084] tablet 3c8200d9c20e4140906c71c746badb94: '0e4622d80cc94f12b492bdd7597a61e1' -> 'e7eada5032fb45c3a5e3196e2b3faaab' move ignored: Not found: Peer with uuid 0e4622d80cc94f12b492bdd7597a61e1 not found in consensus config
I20240501 14:09:46.892016 927 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:09:47.570430 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbe6d56762d84933a7b7ddaf8c314b4f: could not find any suitable replica to move from server a63e0dd1ba2e4a8db8ace9718a700c32 to server 0e4622d80cc94f12b492bdd7597a61e1
W20240501 14:09:47.575618 927 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd036c60a12148d2ada73beb29aedcbf: could not find any suitable replica to move from server 9438789dc61f4501ad1d4ceaeccee458 to server e7eada5032fb45c3a5e3196e2b3faaab
I20240501 14:09:47.680536 927 rebalancer_tool.cc:1074] tablet 4603d1b1b1d441b1b0be0464e2f2c9e2: 'a63e0dd1ba2e4a8db8ace9718a700c32' -> 'b934fd09590a40ada094b9715d990a00' move scheduled
I20240501 14:09:47.769611 927 rebalancer_tool.cc:1136] tablet 8ffdb76b2c5149a18d45bee9581e605a: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '9438789dc61f4501ad1d4ceaeccee458' move completed: Incomplete: tablet 8ffdb76b2c5149a18d45bee9581e605a, TS e7eada5032fb45c3a5e3196e2b3faaab -> TS 9438789dc61f4501ad1d4ceaeccee458 move failed, target replica disappeared
I20240501 14:09:47.936877 927 rebalancer_tool.cc:1136] tablet 9d2c173b2c32494eb16e46a51ca36e31: '30e0b8849e3841ca9ab0de01e719ee4a' -> '9438789dc61f4501ad1d4ceaeccee458' move completed: Incomplete: tablet 9d2c173b2c32494eb16e46a51ca36e31, TS 30e0b8849e3841ca9ab0de01e719ee4a -> TS 9438789dc61f4501ad1d4ceaeccee458 move failed, target replica disappeared
W20240501 14:09:48.159636 4055 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:09:47.955507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.191.70:38181 (local address 127.0.0.1:46462)
0501 14:09:47.955754 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0501 14:09:47.955766 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 14:09:47.955847 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:09:48.159002 (+203155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":61,"threads_started":1}
I20240501 14:09:48.161860 927 rebalancer_tool.cc:1123] tablet c082ccc777ba4f9a937931f707645e54: '6a6fcc099d1340c383e5a429643d5b7c' -> 'b934fd09590a40ada094b9715d990a00' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:09:48.175783 927 rebalancer_tool.cc:1123] tablet ad0dac76fd0c4ae6b8e0549376c98e5d: 'e7eada5032fb45c3a5e3196e2b3faaab' -> 'a0caf2974041408bb32362bfc4284eaa' move is abandoned: Illegal state: ad0dac76fd0c4ae6b8e0549376c98e5d: source replica e7eada5032fb45c3a5e3196e2b3faaab does not have REPLACE attribute set
I20240501 14:09:48.195629 927 rebalancer_tool.cc:1136] tablet bca09920a7904362bc546916e3f6f5ac: '6a6fcc099d1340c383e5a429643d5b7c' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move completed: OK
I20240501 14:09:48.201805 927 rebalancer_tool.cc:1136] tablet 5d994081c3414f779ba2cae431cb1d97: '0e4622d80cc94f12b492bdd7597a61e1' -> 'b934fd09590a40ada094b9715d990a00' move completed: OK
I20240501 14:09:48.241045 927 rebalancer_tool.cc:1136] tablet 72ad9c714b2b4fff94063d2838835173: '6a6fcc099d1340c383e5a429643d5b7c' -> '30e0b8849e3841ca9ab0de01e719ee4a' move completed: OK
I20240501 14:09:48.258739 927 rebalancer_tool.cc:1136] tablet ea66fe3d0da04d18b961927a9c988a20: 'e7eada5032fb45c3a5e3196e2b3faaab' -> '0e4622d80cc94f12b492bdd7597a61e1' move completed: OK
I20240501 14:09:49.454627 927 rebalancer_tool.cc:1136] tablet 4603d1b1b1d441b1b0be0464e2f2c9e2: 'a63e0dd1ba2e4a8db8ace9718a700c32' -> 'b934fd09590a40ada094b9715d990a00' move completed: OK
I20240501 14:14:47.331871 927 rebalancer_tool.cc:1136] tablet b1006490ec2841b2a3365295caaf19e1: 'e7eada5032fb45c3a5e3196e2b3faaab' -> 'a63e0dd1ba2e4a8db8ace9718a700c32' move completed: Incomplete: tablet b1006490ec2841b2a3365295caaf19e1, TS e7eada5032fb45c3a5e3196e2b3faaab -> TS a63e0dd1ba2e4a8db8ace9718a700c32 move failed, target replica disappeared
I20240501 14:14:48.015312 927 rebalancer_tool.cc:1123] tablet 96f5e4e25a674a99a6762757f0ab1545: '0e4622d80cc94f12b492bdd7597a61e1' -> '9438789dc61f4501ad1d4ceaeccee458' move is abandoned: Network error: recv got EOF from 127.23.191.69:45429 (error 108)
I20240501 14:14:48.021159 927 rebalancer_tool.cc:1123] tablet 5b2e30cb9de14fe19273b3cef801cb67: '0e4622d80cc94f12b492bdd7597a61e1' -> 'a0caf2974041408bb32362bfc4284eaa' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:48.021248 927 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:14:48.154125 10669 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:14:48.087010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.191.71:35407 (local address 127.0.0.1:42286)
0501 14:14:48.089704 (+ 2694us) negotiation.cc:107] Waiting for socket to connect
0501 14:14:48.089716 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 14:14:48.090078 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:14:48.154019 (+ 63941us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2643}
Illegal state: tablet server 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): unacceptable health status UNAVAILABLE
I20240501 14:14:48.176867 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session e7eada5032fb45c3a5e3196e2b3faaab-5d994081c3414f779ba2cae431cb1d97 received from {username='slave'} at 127.23.191.68:40875
I20240501 14:14:48.176951 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session e7eada5032fb45c3a5e3196e2b3faaab-5d994081c3414f779ba2cae431cb1d97 on tablet 5d994081c3414f779ba2cae431cb1d97 with peer e7eada5032fb45c3a5e3196e2b3faaab
I20240501 14:14:48.253942 9613 consensus_queue.cc:579] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Leader has been unable to successfully communicate with peer 9438789dc61f4501ad1d4ceaeccee458 for more than 300 seconds (300.096s)
I20240501 14:14:48.256693 31450 consensus_queue.cc:237] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:14:48.258013 31319 raft_consensus.cc:1261] T 7a542c07a3f841b0a31051a8a37083bf P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 FOLLOWER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 14:14:48.258455 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:48.258792 10693 consensus_queue.cc:1035] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:14:48.258996 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.263739 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:48.267745 31450 raft_consensus.cc:1203] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.16->[1.17-1.17] Dedup: 1.17->[]
W20240501 14:14:48.271663 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.283313 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:48.297447 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2e68be1521964ba8b864801fdf3b0df3: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:48.331395 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.340245 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:48.342348 10378 consensus_queue.cc:579] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Leader has been unable to successfully communicate with peer 9438789dc61f4501ad1d4ceaeccee458 for more than 300 seconds (300.343s)
W20240501 14:14:48.353272 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:48.372661 10378 consensus_queue.cc:579] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Leader has been unable to successfully communicate with peer 9438789dc61f4501ad1d4ceaeccee458 for more than 300 seconds (300.439s)
I20240501 14:14:48.373898 31319 raft_consensus.cc:1203] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 14:14:48.376005 31188 consensus_queue.cc:237] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:14:48.378000 31580 raft_consensus.cc:1261] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:14:48.378127 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.378468 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:48.378479 10378 consensus_queue.cc:1035] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:14:48.379145 10716 ts_tablet_manager.cc:902] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c: Initiating tablet copy from peer 30e0b8849e3841ca9ab0de01e719ee4a (127.23.191.66:45265)
I20240501 14:14:48.379606 10716 tablet_copy_client.cc:308] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Beginning tablet copy session from remote peer at address 127.23.191.66:45265
I20240501 14:14:48.380049 31340 tablet_copy_service.cc:140] P 30e0b8849e3841ca9ab0de01e719ee4a: Received BeginTabletCopySession request for tablet f26d16499ab54a3e9059f6e4ff041177 from peer 6a6fcc099d1340c383e5a429643d5b7c ({username='slave'} at 127.23.191.67:41231)
I20240501 14:14:48.380136 31340 tablet_copy_service.cc:161] P 30e0b8849e3841ca9ab0de01e719ee4a: Beginning new tablet copy session on tablet f26d16499ab54a3e9059f6e4ff041177 from peer 6a6fcc099d1340c383e5a429643d5b7c at {username='slave'} at 127.23.191.67:41231: session id = 6a6fcc099d1340c383e5a429643d5b7c-f26d16499ab54a3e9059f6e4ff041177
I20240501 14:14:48.380573 9174 raft_consensus.cc:2932] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:14:48.381021 31340 tablet_copy_source_session.cc:205] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:14:48.381986 31031 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 28013d7df97943e09c27c3bb1794ee80 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:48.382401 31580 raft_consensus.cc:2932] T 28013d7df97943e09c27c3bb1794ee80 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:14:48.382827 10716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26d16499ab54a3e9059f6e4ff041177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:14:48.384418 31046 catalog_manager.cc:5484] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 6 to 7, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "0e4622d80cc94f12b492bdd7597a61e1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:48.385118 10716 tablet_copy_client.cc:783] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Starting download of 5 data blocks...
I20240501 14:14:48.393292 31629 maintenance_manager.cc:419] P e7eada5032fb45c3a5e3196e2b3faaab: Scheduling UndoDeltaBlockGCOp(5d994081c3414f779ba2cae431cb1d97): 4921 bytes on disk
I20240501 14:14:48.393543 10716 tablet_copy_client.cc:647] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:14:48.393911 31519 maintenance_manager.cc:643] P e7eada5032fb45c3a5e3196e2b3faaab: UndoDeltaBlockGCOp(5d994081c3414f779ba2cae431cb1d97) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 14:14:48.395171 10716 tablet_copy_client.cc:523] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:14:48.396643 10716 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap starting.
I20240501 14:14:48.405908 10716 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:14:48.406169 10716 tablet_bootstrap.cc:492] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c: Bootstrap complete.
I20240501 14:14:48.406313 10716 ts_tablet_manager.cc:1367] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.006s
I20240501 14:14:48.406680 10716 raft_consensus.cc:348] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } }
I20240501 14:14:48.406785 10716 raft_consensus.cc:729] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a6fcc099d1340c383e5a429643d5b7c, State: Initialized, Role: LEARNER
I20240501 14:14:48.406864 10716 consensus_queue.cc:260] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: NON_VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: true } }
I20240501 14:14:48.407099 10716 ts_tablet_manager.cc:1398] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:14:48.407560 31340 tablet_copy_service.cc:342] P 30e0b8849e3841ca9ab0de01e719ee4a: Request end of tablet copy session 6a6fcc099d1340c383e5a429643d5b7c-f26d16499ab54a3e9059f6e4ff041177 received from {username='slave'} at 127.23.191.67:41231
I20240501 14:14:48.407632 31340 tablet_copy_service.cc:433] P 30e0b8849e3841ca9ab0de01e719ee4a: ending tablet copy session 6a6fcc099d1340c383e5a429643d5b7c-f26d16499ab54a3e9059f6e4ff041177 on tablet f26d16499ab54a3e9059f6e4ff041177 with peer 6a6fcc099d1340c383e5a429643d5b7c
I20240501 14:14:48.414992 10698 consensus_queue.cc:579] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Leader has been unable to successfully communicate with peer 9438789dc61f4501ad1d4ceaeccee458 for more than 300 seconds (300.512s)
I20240501 14:14:48.417474 31580 consensus_queue.cc:237] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:14:48.418956 31450 raft_consensus.cc:1261] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:14:48.419281 10697 consensus_queue.cc:1035] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:14:48.419656 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.419694 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:48.420992 10390 raft_consensus.cc:2932] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:14:48.421427 31450 raft_consensus.cc:2932] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:14:48.422358 31033 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6bcd6a2f3c014d5ca0e32e16cdbc4edc with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:48.423429 31041 catalog_manager.cc:5484] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab reported cstate change: config changed from index 6 to 7, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:14:48.459683 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:48.527767 31498 maintenance_manager.cc:419] P 6a6fcc099d1340c383e5a429643d5b7c: Scheduling UndoDeltaBlockGCOp(f26d16499ab54a3e9059f6e4ff041177): 4579 bytes on disk
I20240501 14:14:48.528218 31388 maintenance_manager.cc:643] P 6a6fcc099d1340c383e5a429643d5b7c: UndoDeltaBlockGCOp(f26d16499ab54a3e9059f6e4ff041177) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240501 14:14:48.597868 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:14:48.610502 10731 consensus_queue.cc:579] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Leader has been unable to successfully communicate with peer a63e0dd1ba2e4a8db8ace9718a700c32 for more than 300 seconds (300.426s)
I20240501 14:14:48.642104 9174 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:48.657462 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:48.670533 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:48.680053 31580 raft_consensus.cc:1203] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240501 14:14:48.724121 10731 consensus_queue.cc:579] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Leader has been unable to successfully communicate with peer 9438789dc61f4501ad1d4ceaeccee458 for more than 300 seconds (300.540s)
I20240501 14:14:48.729328 9174 raft_consensus.cc:1050] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1: attempting to promote NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c to VOTER
I20240501 14:14:48.729728 9174 consensus_queue.cc:237] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false replace: true } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } }
W20240501 14:14:48.730281 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:48.731112 31450 raft_consensus.cc:1261] T b1006490ec2841b2a3365295caaf19e1 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Refusing update from remote peer 0e4622d80cc94f12b492bdd7597a61e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
W20240501 14:14:48.731523 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:48.731556 10748 consensus_queue.cc:1035] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20240501 14:14:48.731703 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.791961 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:14:48.898995 9613 raft_consensus.cc:1050] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c: attempting to promote NON_VOTER 30e0b8849e3841ca9ab0de01e719ee4a to VOTER
I20240501 14:14:48.899412 9613 consensus_queue.cc:237] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false } }
I20240501 14:14:48.900636 31319 raft_consensus.cc:1261] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEARNER]: Refusing update from remote peer 6a6fcc099d1340c383e5a429643d5b7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:14:48.901244 10766 consensus_queue.cc:1035] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 14:14:48.901458 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:48.901592 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:49.000545 31450 raft_consensus.cc:1203] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:14:49.018234 10697 consensus_queue.cc:579] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Leader has been unable to successfully communicate with peer a63e0dd1ba2e4a8db8ace9718a700c32 for more than 300 seconds (300.062s)
I20240501 14:14:49.021833 31580 consensus_queue.cc:237] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } }
I20240501 14:14:49.022215 10748 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:14:49.022330 10748 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:14:49.022727 10748 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:14:49.023211 31188 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:14:49.023206 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:14:49.023646 10698 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:14:49.023759 31450 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:14:49.024279 10698 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:14:49.024644 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:49.024720 31124 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: b934fd09590a40ada094b9715d990a00, e7eada5032fb45c3a5e3196e2b3faaab
I20240501 14:14:49.024890 10748 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:14:49.025470 10697 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:14:49.025872 31188 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:14:49.027108 31033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d265eff200b645bfac8f8906654385f6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 14:14:49.027817 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:49.028419 31043 catalog_manager.cc:5484] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 8 to 9, VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:14:49.029320 31450 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:14:49.033017 31580 consensus_queue.cc:237] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } }
I20240501 14:14:49.034289 31188 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:14:49.034294 31450 raft_consensus.cc:1261] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Refusing update from remote peer e7eada5032fb45c3a5e3196e2b3faaab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:14:49.034762 10697 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:14:49.035264 10390 consensus_queue.cc:1035] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:14:49.035696 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:49.036386 10390 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:14:49.036743 31188 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:14:49.037465 31450 raft_consensus.cc:2932] T d265eff200b645bfac8f8906654385f6 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
I20240501 14:14:49.037909 31033 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d265eff200b645bfac8f8906654385f6 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:49.039130 31046 catalog_manager.cc:5484] T d265eff200b645bfac8f8906654385f6 P 0e4622d80cc94f12b492bdd7597a61e1 reported cstate change: config changed from index 9 to 10, NON_VOTER b934fd09590a40ada094b9715d990a00 (127.23.191.69) added. New cstate: current_term: 1 leader_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: NON_VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: true } } }
W20240501 14:14:49.043632 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:49.078694 10698 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:14:49.078859 10698 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:14:49.079334 10698 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:14:49.079732 31188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:14:49.079895 31188 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:14:49.080284 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:49.080384 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:49.080461 31518 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:14:49.080598 10698 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:49.136638 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 14:14:49.155400 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:14:49.161737 10748 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:14:49.161885 10748 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:14:49.162354 10748 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:14:49.163213 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:49.163321 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:14:49.163399 31123 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a0caf2974041408bb32362bfc4284eaa, a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:14:49.163463 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:49.163586 10748 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:14:49.217876 10731 raft_consensus.cc:1050] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER e7eada5032fb45c3a5e3196e2b3faaab to VOTER
I20240501 14:14:49.218390 10731 consensus_queue.cc:237] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false } }
W20240501 14:14:49.220930 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:49.220940 31580 raft_consensus.cc:1261] T 5d994081c3414f779ba2cae431cb1d97 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:14:49.221462 10802 consensus_queue.cc:1035] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 14:14:49.221854 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:49.255856 9861 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:49.279830 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:49.307802 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:49.318140 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d265eff200b645bfac8f8906654385f6: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:49.320535 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:49.434641 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:14:49.503368 31036 maintenance_manager.cc:419] P d913062e66274883a568f59820e333ee: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 11878 bytes on disk
I20240501 14:14:49.505885 31035 maintenance_manager.cc:643] P d913062e66274883a568f59820e333ee: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.002s sys 0.000s Metrics: {"dirs.queue_time_us":190,"thread_start_us":78,"threads_started":1}
W20240501 14:14:49.561976 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:14:49.602393 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:49.605893 10802 raft_consensus.cc:1050] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a: attempting to promote NON_VOTER 6a6fcc099d1340c383e5a429643d5b7c to VOTER
I20240501 14:14:49.606344 10802 consensus_queue.cc:237] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } }
I20240501 14:14:49.607700 31580 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:14:49.607945 31450 raft_consensus.cc:1261] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 LEARNER]: Refusing update from remote peer 30e0b8849e3841ca9ab0de01e719ee4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:14:49.608429 10731 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:14:49.608595 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:49.608613 10505 consensus_queue.cc:1035] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:14:49.610360 10731 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:14:49.610823 31580 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P e7eada5032fb45c3a5e3196e2b3faaab [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:14:49.612716 31450 raft_consensus.cc:2932] T f26d16499ab54a3e9059f6e4ff041177 P 6a6fcc099d1340c383e5a429643d5b7c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } } }
I20240501 14:14:49.613373 31043 catalog_manager.cc:5484] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a reported cstate change: config changed from index 9 to 10, 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:14:49.635751 9861 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:49.692154 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:49.725788 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:49.726831 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:49.813315 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240501 14:14:49.896343 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:49.934700 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:14:50.055826 10748 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:50.143654 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:50.207573 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:50.314913 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:50.326653 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:50.347043 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.390542 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.402951 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.409540 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.500406 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:14:50.514170 9861 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:50.517658 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.533244 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.566143 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:50.590138 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.599746 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.604365 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.616189 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.644989 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.647912 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.663228 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.718972 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.720018 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.750160 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.765429 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:50.809317 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:50.855911 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:50.864549 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240501 14:14:50.876235 10851 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:14:50.876370 10851 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:14:50.876849 10851 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:14:50.877183 31188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:14:50.877338 31188 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:14:50.877936 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:50.878325 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:50.878386 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:14:50.878396 31518 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:14:50.878561 10851 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:14:50.937961 9861 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:51.006677 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240501 14:14:51.056500 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:51.057262 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:51.090498 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:14:51.096596 9861 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:14:51.096740 9861 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:14:51.097242 9861 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:14:51.098234 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:51.098330 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:14:51.098418 31123 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a0caf2974041408bb32362bfc4284eaa, a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:14:51.098500 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:51.098544 9861 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:51.130043 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:51.135036 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:51.140724 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:51.168263 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:51.215360 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:51.223409 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d265eff200b645bfac8f8906654385f6: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:51.230863 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:51.263595 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:51.308859 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:51.350865 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:51.405483 9861 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:51.440970 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:51.458186 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240501 14:14:51.670083 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240501 14:14:51.670655 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240501 14:14:51.672677 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:51.752525 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:51.806684 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:51.831418 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:51.871047 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:51.929533 11010 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:51.956481 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:14:51.971613 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:51.990763 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:52.138108 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240501 14:14:52.175875 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:52.211570 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:52.276077 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240501 14:14:52.376313 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 14:14:52.376968 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:52.384196 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:52.483995 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:14:52.491886 9861 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:14:52.612123 10963 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:14:52.612262 10963 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:14:52.612746 10963 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:14:52.613339 31188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:14:52.613518 31188 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:14:52.614025 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:52.614104 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:52.614190 31518 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:14:52.614477 10963 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:52.647125 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:52.688889 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240501 14:14:52.725131 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:52.732509 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:52.797567 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:52.888425 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:52.928694 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:52.943642 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:52.957681 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:52.969899 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:52.976339 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:53.002293 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.012027 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.078475 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.090251 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.096251 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240501 14:14:53.096786 9861 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:53.099953 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.134184 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:53.152805 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.195609 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.248677 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.274392 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:53.289996 31034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:14:53.290693 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d265eff200b645bfac8f8906654385f6: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:53.305405 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:53.335798 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.383491 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:53.393549 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.393595 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.424604 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.425485 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:14:53.512744 11099 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:53.579203 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 14:14:53.599226 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.622838 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.629356 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.634095 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.691871 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.710628 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:53.730628 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:53.767513 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.788961 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:53.818825 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.879956 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:53.896647 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:53.966003 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240501 14:14:54.025913 9861 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:14:54.026048 9861 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:14:54.026535 9861 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:14:54.027509 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:54.027607 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:14:54.027678 31123 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a0caf2974041408bb32362bfc4284eaa, b934fd09590a40ada094b9715d990a00
W20240501 14:14:54.027752 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:54.027815 9861 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:14:54.031092 11173 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:54.223510 31386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:14:54.294514 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:54.295619 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:54.333976 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:54.400173 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:54.409044 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6f8bd03c05d84fdaaf138a6d0608e30e: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:54.422761 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:54.435009 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:14:54.468070 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:54.493686 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 14:14:54.596499 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:14:54.638491 11173 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:54.705581 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:54.761821 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 14:14:54.779129 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:54.804720 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:54.831485 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 14:14:54.888422 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:54.930534 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:14:54.962057 11173 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:14:54.962182 11173 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:14:54.962616 11173 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:14:54.963081 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:14:54.963235 31580 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:14:54.963840 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:54.964056 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:54.964128 31125 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:14:54.964274 11173 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:14:55.226140 11173 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:55.233537 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:55.246610 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.259781 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:55.264066 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:55.286197 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:55.313251 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.442487 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.462201 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.463546 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 14:14:55.648209 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:55.660372 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.670410 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.670522 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.671025 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b1006490ec2841b2a3365295caaf19e1: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:55.679332 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.685431 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:55.685602 11269 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:55.686589 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.686805 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.749241 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.792476 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:55.828322 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.856702 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.857617 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.875672 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:55.893462 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.912086 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.920536 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:55.969065 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:56.006839 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:56.020543 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:56.030793 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:56.050863 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:56.094826 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:56.102209 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:56.153484 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 14:14:56.171872 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:56.208446 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:56.212217 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:56.271528 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:56.289805 11173 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:56.339367 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:56.370100 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:56.423508 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 14:14:56.437987 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:56.532145 31124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240501 14:14:56.654979 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:56.705174 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:56.711925 31255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 14:14:56.712520 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:56.715483 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:56.734453 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 14:14:56.734566 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:56.791795 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:56.811558 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:56.906014 11173 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:57.004709 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:14:57.211694 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:57.221468 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:57.241873 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 14:14:57.286219 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:57.305238 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:14:57.308615 11349 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:57.344693 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 14:14:57.406396 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d265eff200b645bfac8f8906654385f6: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:57.502988 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:57.587150 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:57.643492 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:57.719317 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:57.734009 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240501 14:14:57.759518 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:57.805181 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:57.817970 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 14:14:57.828982 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:57.883381 11173 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:57.891628 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.003700 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.068670 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:58.084899 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:14:58.120095 11173 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:14:58.120244 11173 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:14:58.120726 11173 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:14:58.121158 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:14:58.121333 31580 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:14:58.121975 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:14:58.122092 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:58.122156 31124 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:14:58.122287 11173 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:58.146369 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.175472 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.192644 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.201355 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.219206 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.222631 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.228286 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:58.325537 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.343096 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.343309 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.368237 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:58.380537 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.389561 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:58.453637 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:14:58.464138 11508 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:14:58.472016 11173 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:14:58.472154 11173 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:14:58.472635 11173 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:14:58.473701 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:58.473834 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:14:58.473904 31123 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a0caf2974041408bb32362bfc4284eaa, b934fd09590a40ada094b9715d990a00
W20240501 14:14:58.473970 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:14:58.474035 11173 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:58.506438 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.521474 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.533434 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.552737 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.586558 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.630611 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:58.632624 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.709576 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:58.712541 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.772616 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.805634 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:58.818751 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.820879 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:58.831585 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:58.861634 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 14:14:58.878989 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 14:14:58.942304 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:59.027117 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 14:14:59.063172 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:14:59.066444 11557 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:14:59.164597 11350 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:59.164724 11350 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:14:59.165202 11350 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:14:59.166162 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:14:59.166255 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:59.166353 31517 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
W20240501 14:14:59.166433 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:14:59.166497 11350 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:59.171032 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:59.185674 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:59.215252 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:59.244416 31386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:14:59.363484 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:59.376173 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:59.382920 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:59.578630 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:59.616478 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:14:59.658972 11589 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:14:59.672103 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 14:14:59.682018 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:14:59.706419 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:14:59.803360 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:14:59.805791 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 14:15:00.028460 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.145054 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.173678 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:00.188695 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.221237 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:00.265379 11651 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:00.273558 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:00.351109 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 14:15:00.370633 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.415418 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:00.441524 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.450748 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.541188 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.559710 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.602524 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.636037 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:00.683496 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:00.716431 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.727903 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.789000 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.826810 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:00.865952 11689 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:00.881167 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.926962 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:15:00.935150 11689 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:00.935272 11689 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:00.935720 11689 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:15:00.936147 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:15:00.936298 31580 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:15:00.936749 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.936857 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:15:00.936985 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:00.937055 31124 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:15:00.937191 11689 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:00.945489 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:00.970125 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:01.024343 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:01.042750 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:01.082587 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:01.088658 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:01.112391 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:01.124708 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:01.136255 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:01.175351 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:01.220307 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:01.245362 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:01.283612 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:01.334965 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:15:01.344252 11689 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:01.344377 11689 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:15:01.344815 11689 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:15:01.345754 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:15:01.345837 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:01.345888 31123 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a0caf2974041408bb32362bfc4284eaa, a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:15:01.345964 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:01.346022 11689 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:01.358578 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:01.363559 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240501 14:15:01.371798 11689 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:01.382918 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:01.386454 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 14:15:01.428946 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 14:15:01.438975 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:01.548013 31123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:15:01.666977 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:01.685483 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:01.721745 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:01.732466 31254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 14:15:01.750164 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:01.794318 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:01.821012 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:01.966113 11689 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:01.986131 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:02.094223 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 14:15:02.148977 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 14:15:02.154522 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:02.201272 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:02.225730 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:15:02.284803 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:02.301908 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:02.358022 11780 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:02.359484 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:02.379449 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:02.738569 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:02.745656 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 14:15:02.746205 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:02.783540 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:02.795167 11780 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:02.795291 11780 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:02.795791 11780 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:15:02.796020 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
W20240501 14:15:02.796038 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:02.796195 31580 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:15:02.796875 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:02.797004 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:15:02.797072 31125 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:15:02.797187 11780 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:02.833542 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:02.843581 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:02.917474 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240501 14:15:02.921711 11849 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:02.923218 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:02.946784 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:02.955673 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:02.964237 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.010244 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.194571 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.223999 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:03.274423 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.306294 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:03.319492 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.323318 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:15:03.340802 11780 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:03.355932 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.362871 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.366024 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:03.426003 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.495340 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.495482 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:03.500185 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.502007 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.518970 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.601795 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.616019 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.624518 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.625741 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.635653 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:03.709738 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.740613 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.788753 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:03.830878 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 14:15:03.872143 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:03.879779 31034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:15:03.880421 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b1006490ec2841b2a3365295caaf19e1: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:15:03.885381 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:03.901932 11849 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:03.919237 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 14:15:03.979205 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:04.002317 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:04.071493 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 14:15:04.187083 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:04.246167 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:04.249668 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:04.284767 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:04.290103 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:04.371693 11849 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:04.380509 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 14:15:04.426648 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 14:15:04.499639 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:04.565217 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:04.589731 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:04.598399 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:15:04.601578 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 14:15:04.728698 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:04.807669 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:04.808036 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:04.859100 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:04.960436 11849 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:05.105374 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:05.122020 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.125207 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.210050 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:05.217550 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:05.363788 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.408210 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.431744 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 14:15:05.431833 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:05.473659 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.498193 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:15:05.502979 11849 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:05.505103 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.571146 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.622159 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d265eff200b645bfac8f8906654385f6: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:15:05.697574 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:05.709487 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.719216 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:05.777794 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:05.808938 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:05.811389 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.818579 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.867179 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.880738 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.886126 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.919503 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.943652 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:05.990713 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:15:06.020304 12096 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:06.026304 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:06.031613 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:06.032547 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:06.162305 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:06.165014 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:06.185564 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:06.239104 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:06.294876 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:06.356698 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:06.362223 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:06.367031 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 14:15:06.390318 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:06.452337 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:06.471078 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 14:15:06.472075 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240501 14:15:06.534844 12132 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:06.543021 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:06.573657 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:06.586002 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 14:15:06.590090 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:06.594233 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:15:06.623044 12132 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:06.623229 12132 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:15:06.623771 12132 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:15:06.624977 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:06.625277 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:06.625342 31123 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a0caf2974041408bb32362bfc4284eaa, b934fd09590a40ada094b9715d990a00
W20240501 14:15:06.625430 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:15:06.625725 12132 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:06.745189 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:06.749284 31255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 14:15:06.778992 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 14:15:06.791489 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:06.802681 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:07.026917 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:07.048437 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 14:15:07.076346 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:15:07.095528 12132 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:15:07.101361 12159 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:07.101492 12159 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:07.101992 12159 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:15:07.102370 31188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:15:07.102545 31188 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:15:07.103009 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:07.103099 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:15:07.103180 31518 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:15:07.103350 12159 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:07.182725 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:07.256458 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:07.306514 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:07.407959 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:07.410482 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:07.509297 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:15:07.559744 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:07.562844 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:07.571498 12177 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:07.666363 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:07.691314 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:07.791656 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240501 14:15:07.878859 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:07.894716 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:07.920498 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:07.947324 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.017699 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 14:15:08.032980 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:08.033985 12132 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:08.068279 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.083240 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.113451 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:08.130195 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:08.152529 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.172552 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.186710 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.209017 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.215327 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.220147 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.273322 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.322207 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:08.326169 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.327736 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.339919 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.492980 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.524976 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:08.548282 12288 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:08.558671 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.592322 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:08.642750 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.662395 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.715711 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.752765 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:08.807614 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:08.838354 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.841548 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.853698 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:08.861230 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.875751 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240501 14:15:08.937156 12288 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:08.939108 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.948477 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:08.973655 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 14:15:08.973791 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:09.001255 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 14:15:09.057318 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:09.146050 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240501 14:15:09.170037 12159 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:09.170164 12159 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:09.170650 12159 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:15:09.170923 31188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:15:09.171073 31188 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:15:09.171736 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:09.172272 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:15:09.172344 31518 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:15:09.172523 12159 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:09.173202 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:09.349363 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:15:09.376096 12288 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:15:09.381145 12159 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:15:09.381260 12159 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:09.381696 12159 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:15:09.382748 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:15:09.382843 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:09.382896 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:15:09.382931 31517 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 1 yes votes; 3 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
I20240501 14:15:09.383061 12159 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:09.410854 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:09.509159 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:09.520251 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 14:15:09.568773 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:09.673990 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:09.685180 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 14:15:09.764058 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:09.788849 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:09.933964 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:15:09.948175 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:09.969369 12288 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:09.971414 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:10.017491 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:10.181826 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:10.192440 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:10.256574 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:10.406035 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:10.406265 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:10.413561 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:10.420171 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:10.443115 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:10.446260 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:15:10.456765 12288 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:10.517668 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:10.548123 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 14:15:10.555586 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:10.591858 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:10.655355 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:10.673503 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:10.703747 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:10.707157 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:10.719744 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:10.787814 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:10.808436 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:15:10.836135 12132 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:10.836766 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:10.852763 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:10.976343 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:10.989655 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:11.029906 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:11.051093 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:11.052292 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:11.091360 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:11.129624 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:11.177202 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:11.228307 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:15:11.237494 12288 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:11.311807 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:11.328405 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:11.349728 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 14:15:11.437994 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:11.449285 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:11.489735 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 14:15:11.489755 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:11.500190 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:11.511229 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 14:15:11.522686 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:11.523806 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:11.599162 31123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 14:15:11.635062 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:11.694713 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 14:15:11.696944 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:11.761197 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 119 similar messages]
I20240501 14:15:11.847643 12533 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:11.865149 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:11.949193 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:11.987866 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:12.110821 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:12.141736 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 14:15:12.157279 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:12.317888 12288 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:12.337257 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:12.374754 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:12.442045 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:12.442538 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:12.469565 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:12.554579 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:15:12.572110 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:12.642566 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:12.671743 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:12.733577 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:12.804522 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240501 14:15:12.805181 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:12.821112 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 14:15:12.874508 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:15:12.879956 12288 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:15:12.913641 12288 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:12.913794 12288 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:12.914382 12288 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:15:12.914858 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:15:12.915009 31580 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:15:12.915483 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:15:12.916029 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:12.916110 31124 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:15:12.916267 12288 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:12.973423 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:12.994801 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.023602 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.047313 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.066217 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.086494 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:13.120244 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:13.134938 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:13.136458 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.162695 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:13.211426 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.277222 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.286175 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.300935 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.361757 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.430615 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:15:13.457408 12657 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:13.487883 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.517361 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.546136 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:13.549309 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.596534 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:13.606143 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:13.625775 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.694810 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.699707 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:13.735394 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:13.797698 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.803170 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 14:15:13.812253 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.814771 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 14:15:13.824707 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:13.866482 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240501 14:15:14.010277 12288 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:14.087817 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:14.087822 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:14.123059 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:14.125769 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 14:15:14.126668 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:14.217804 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:14.238042 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:14.262552 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:14.301776 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:14.502231 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:14.521883 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:15:14.547674 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:14.624365 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240501 14:15:14.631223 12288 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:14.765167 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:14.821115 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:14.839398 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:14.917280 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:15.015650 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:15.031338 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.102480 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.118737 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:15:15.132450 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:15.174644 12288 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:15.206791 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:15.223214 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:15.330727 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 14:15:15.338079 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.417125 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.431095 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.509480 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.511742 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.513857 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.522060 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.528329 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.568997 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.595429 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:15.616381 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.650314 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.656466 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:15.666730 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:15.787211 12831 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:15.822274 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:15.843802 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.918514 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.922499 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:15.924329 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:16.023284 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:16.123399 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:16.141307 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:16.191037 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:16.198014 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 14:15:16.204284 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:16.207974 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:16.218031 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:15:16.260802 12288 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:16.262921 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 14:15:16.275979 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:16.280710 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:16.298065 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:16.319118 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:16.372912 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 14:15:16.411347 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:16.490796 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:16.534696 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:16.677209 31124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 14:15:16.702322 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:16.720783 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 14:15:16.724210 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:16.726132 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:16.747367 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:16.762018 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 121 similar messages]
I20240501 14:15:16.763393 12288 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:16.813762 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:16.871337 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:17.092365 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 14:15:17.185271 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:17.286634 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:17.289620 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:17.320396 12945 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:17.340044 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:17.349421 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:17.488653 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:15:17.528440 12945 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:17.528573 12945 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:15:17.528999 12945 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:15:17.529990 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:15:17.530077 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:17.530125 31124 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
I20240501 14:15:17.530252 12945 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:17.530439 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:17.619522 12913 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:17.619709 12913 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:17.620230 12913 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:15:17.620682 31188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:15:17.620863 31188 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:15:17.621449 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:15:17.621631 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:17.621712 31517 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:15:17.621842 12913 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:17.630870 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:17.665653 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:17.687150 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:17.700783 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:17.709416 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:15:17.795665 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:15:17.801048 12945 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:17.808272 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:17.820806 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 14:15:17.821491 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:17.824077 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:17.873980 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:17.900216 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:17.964651 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:18.055533 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.071666 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.119796 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.179174 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.184355 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:18.185278 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.190627 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.190917 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240501 14:15:18.251793 12945 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:18.257710 31032 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: ChangeConfig RPC to 127.23.191.67:39413 timed out after 30.000s (SENT) [suppressed 6 similar messages]
W20240501 14:15:18.258957 31032 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 7a542c07a3f841b0a31051a8a37083bf with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
W20240501 14:15:18.289866 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:18.319020 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:18.340816 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.368846 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.386004 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.397375 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.430374 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.526728 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:18.587049 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.611477 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.701506 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:18.731267 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.731268 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.749878 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 14:15:18.770006 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:18.795652 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.843710 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240501 14:15:18.861888 13061 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:18.865481 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:18.899538 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:18.937026 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:18.976696 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:19.037036 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 14:15:19.073280 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 14:15:19.087198 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:19.174481 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:19.291857 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:19.294153 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:19.314685 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:19.329785 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:19.353292 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:19.372682 13107 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:19.466969 31032 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 7a542c07a3f841b0a31051a8a37083bf with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
W20240501 14:15:19.581568 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 14:15:19.632912 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:19.661417 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 14:15:19.768888 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:19.806501 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:19.816962 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:19.884534 13061 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:19.892701 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:20.011937 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.106488 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:20.135790 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.139869 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.156556 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:20.173275 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:15:20.177734 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.183190 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.191852 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.297183 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:15:20.299762 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:20.301764 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:20.343226 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:15:20.352560 13107 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:20.380757 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:20.537922 31032 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 7a542c07a3f841b0a31051a8a37083bf with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
W20240501 14:15:20.567222 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.587219 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:20.594089 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.597605 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.660100 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.661370 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.675890 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.701129 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.751618 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.762864 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 14:15:20.763947 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:20.826112 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.836519 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.843202 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:20.867353 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:20.883199 13061 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:20.918015 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:21.000026 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:21.030213 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:21.083719 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 14:15:21.103523 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:21.131930 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:21.152302 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:21.204385 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:21.315044 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:21.375504 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:21.388166 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240501 14:15:21.391192 13253 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:21.392302 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:21.413185 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:21.532217 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:21.536753 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 14:15:21.574802 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:21.665210 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:21.681622 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:21.754235 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 14:15:21.769064 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240501 14:15:21.798193 31254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 14:15:21.833730 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:21.857208 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:15:21.891093 13061 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:21.898550 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:21.914000 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:22.003365 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 14:15:22.016693 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d265eff200b645bfac8f8906654385f6: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:15:22.075322 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:22.109678 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:22.129223 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:15:22.287232 13061 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:22.314507 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:22.317850 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:22.333717 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:22.440865 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:22.450031 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:22.486308 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:22.608386 31032 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 7a542c07a3f841b0a31051a8a37083bf with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
W20240501 14:15:22.662827 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:22.671931 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:22.681244 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:22.753329 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:15:22.791482 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:22.824779 31516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 14:15:22.825381 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:22.829146 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:22.839752 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:15:22.867606 13253 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:22.899694 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:22.906306 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:22.924233 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:22.946393 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:22.990280 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:22.996224 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:23.032685 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 14:15:23.066860 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:23.125576 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:23.140117 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:23.192239 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:23.241192 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:15:23.301872 13169 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:15:23.302026 13169 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:23.302500 13169 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:15:23.303573 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:15:23.303685 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:15:23.303735 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:15:23.303820 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:23.303890 13169 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:23.365625 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:23.442807 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:23.447180 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:23.448354 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:23.459120 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240501 14:15:23.482116 13253 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:23.488262 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:23.518358 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:23.524358 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:23.608896 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:23.618367 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:23.643748 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:23.662019 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:23.666776 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:23.704607 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:23.859424 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:23.866222 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:23.883652 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 14:15:23.903007 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 14:15:23.923084 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:23.946712 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:23.993011 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:24.065222 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:15:24.096933 13253 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:24.186190 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:24.258617 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 14:15:24.286834 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:24.299293 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:24.355216 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:24.519423 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 14:15:24.559782 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:24.562969 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:24.671561 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 14:15:24.672173 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:15:24.687426 13504 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:24.718919 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:24.775513 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:24.825577 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:24.890833 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:24.961352 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:24.996646 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:25.046715 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:15:25.081446 13253 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:25.122996 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:25.135421 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:15:25.146343 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.154302 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.177816 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.198601 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:25.226263 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.238296 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.240345 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.243075 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:25.273550 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:15:25.326700 13169 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:15:25.326836 13169 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:25.327276 13169 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:15:25.328282 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:25.328400 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:15:25.328460 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
I20240501 14:15:25.328581 13169 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:25.328591 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:15:25.409574 13253 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:25.409722 13253 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:25.410223 13253 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:15:25.410629 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:15:25.410794 31580 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:15:25.411350 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:25.411440 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:15:25.411499 31125 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:15:25.411619 13253 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:25.453274 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 14:15:25.480182 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.489050 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.503196 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:25.576218 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.578259 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:25.621263 13568 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:25.622725 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.642997 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.750722 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.768594 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.775092 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:25.821976 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:25.874914 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.941135 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:25.952005 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:25.997301 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:26.009793 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 14:15:26.014866 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:26.040551 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:26.093665 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:26.106144 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:26.159682 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 14:15:26.166802 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:26.186074 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:26.194901 13568 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:26.225842 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:26.273289 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:26.370997 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:26.418958 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:26.428561 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:26.564816 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:26.598291 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240501 14:15:26.658310 13616 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:26.715898 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:26.747598 31032 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 7a542c07a3f841b0a31051a8a37083bf with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
W20240501 14:15:26.765837 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:26.791980 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:26.835176 31255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240501 14:15:26.835739 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:26.860687 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240501 14:15:26.882993 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:27.094669 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 14:15:27.099586 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:27.130023 13616 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:27.158591 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:27.239142 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 14:15:27.251871 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:27.253756 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:27.312237 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:27.345978 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:27.353924 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:27.491446 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:27.536185 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:27.560741 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:27.564337 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:15:27.571888 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:27.612926 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:27.669580 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:15:27.673545 13568 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:27.720804 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:27.737628 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:27.742290 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:27.795230 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:27.799053 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:27.851022 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:27.854005 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240501 14:15:27.857533 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:27.887430 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:27.917009 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:28.014103 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:15:28.039825 13616 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:28.039983 13616 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:28.040562 13616 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:15:28.041072 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:15:28.041229 31580 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:15:28.041965 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:15:28.042045 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:28.042094 31124 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:15:28.042500 13616 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:28.064802 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 14:15:28.115271 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:28.136337 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:28.152693 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:28.161371 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:15:28.205789 13616 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:28.246618 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 14:15:28.271845 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:28.361032 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:28.364238 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 14:15:28.390297 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:28.413648 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:28.437366 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:28.475163 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:28.479576 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:28.504812 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:28.554826 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:28.624886 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:28.711712 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:28.772797 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:28.816296 13568 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:28.831276 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:28.838321 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:28.838321 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:28.866245 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:28.882388 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:28.906036 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:29.042734 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:29.106221 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 14:15:29.274801 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:29.289498 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:29.301210 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:29.406060 13568 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:29.416263 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:29.431336 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:29.477764 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 14:15:29.517216 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:29.689864 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 14:15:29.751385 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:29.779349 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:29.811096 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:29.860803 13616 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:29.905050 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:29.951571 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:29.967608 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:29.990942 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 14:15:30.051662 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:30.075791 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:30.090188 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:30.094167 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:30.100373 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:30.106356 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:30.115665 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:15:30.159848 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:30.184064 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:30.184159 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:30.247642 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:30.251955 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:30.269131 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:30.404728 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 14:15:30.413988 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:15:30.429351 13926 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:30.469797 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:30.482877 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:30.500284 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:30.509804 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:30.567375 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:30.570525 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:30.609449 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 14:15:30.651288 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:30.681824 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:30.727952 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:15:30.810938 13926 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:30.875196 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:30.882252 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:30.897293 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:31.022143 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:31.044445 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:31.069844 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 14:15:31.173457 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:31.182032 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:15:31.189010 13616 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:31.230254 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:31.263517 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:31.265792 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:31.298943 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:31.397796 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:31.423902 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:31.470489 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:31.475739 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 14:15:31.537295 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:31.578613 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:31.665815 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:31.700973 14023 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:31.720294 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:31.745981 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:31.846748 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 14:15:31.870261 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:31.871461 31124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 14:15:31.899538 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:31.959861 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 14:15:31.988622 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:32.112177 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:32.134266 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:32.171617 14023 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:32.252476 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:32.304176 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:32.316476 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 14:15:32.355674 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:32.462242 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:32.479663 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:32.512543 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:15:32.533922 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:32.561352 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:32.562819 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:32.574584 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:32.596714 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:15:32.608129 14023 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:32.629678 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:32.669246 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:32.688459 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:32.743137 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:32.747000 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:32.784997 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:32.835101 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:32.835784 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:32.842326 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 14:15:32.872018 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 14:15:32.888942 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:32.907213 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 14:15:32.930135 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:32.962404 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:32.987962 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:32.997319 13616 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:33.003391 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:33.141543 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:33.188242 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:33.194902 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:15:33.230402 13715 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:15:33.230547 13715 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:33.231066 13715 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:15:33.232164 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:15:33.232466 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:33.232523 31517 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
W20240501 14:15:33.232585 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:15:33.232640 13715 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:33.298797 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:33.318801 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:33.385102 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:15:33.428033 13616 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:33.504572 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:33.524897 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 14:15:33.547930 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:33.553750 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:33.563424 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:33.661212 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:33.693948 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:33.747021 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:33.761932 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:33.867877 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:33.888252 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:33.888427 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:33.921375 14023 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:33.930825 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:33.940536 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:33.956986 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 14:15:34.081271 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:34.086208 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:34.162957 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:34.289149 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:34.296941 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:34.381405 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:34.406111 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:15:34.427454 14226 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:34.437402 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:34.446704 13715 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:34.446846 13715 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:34.447397 13715 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:15:34.447854 31188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:15:34.448006 31188 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:15:34.448493 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:15:34.448999 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:34.449064 31517 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:15:34.449208 13715 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:34.500741 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:34.583125 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 14:15:34.690780 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:34.750990 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 14:15:34.796167 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:34.802035 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 14:15:34.823668 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:15:34.836333 14226 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:34.860477 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:34.933291 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:34.936254 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:34.941869 31032 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 7a542c07a3f841b0a31051a8a37083bf with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
W20240501 14:15:34.958446 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:34.958882 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:15:34.969676 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.136701 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.165315 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 14:15:35.168821 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:35.199505 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.204011 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.249265 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.256778 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:35.262655 14226 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:35.267900 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.276542 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.302115 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:35.325925 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.341439 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:35.350952 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 14:15:35.375698 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.403609 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.465071 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.489598 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.585932 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:35.615551 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.634351 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.763732 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.776547 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:35.787537 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:35.827188 14226 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:35.827340 14226 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:15:35.827864 14226 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:15:35.828745 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:35.828866 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:15:35.828943 31125 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
I20240501 14:15:35.829089 14226 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:35.829239 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:35.861503 14226 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:35.932755 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:35.955617 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:35.984642 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:35.987955 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 14:15:36.085021 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:36.105960 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:36.160326 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:36.213616 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:36.231664 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:15:36.241287 14226 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:36.298818 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:36.329095 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:36.345279 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:36.395866 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 14:15:36.416373 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:36.438066 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:36.488742 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:15:36.616640 14327 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:36.620214 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:36.653086 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:36.704912 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:36.710250 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:36.728874 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:36.822216 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 14:15:36.824203 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:36.880030 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 14:15:36.921452 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 14:15:36.962571 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:36.964820 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:37.000003 14327 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:37.112018 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:37.267726 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:37.269464 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:37.293120 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 14:15:37.350011 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:37.390673 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:37.419996 14327 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:37.427044 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:37.461107 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:37.513307 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:37.516888 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 14:15:37.562669 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:37.590180 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 14:15:37.615803 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:37.638085 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:15:37.664974 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:37.785270 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:37.809476 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:37.867403 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:37.880185 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240501 14:15:37.880782 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:37.887419 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:15:37.898015 14226 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:37.930263 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:37.939548 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:37.971900 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:38.018980 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:38.023720 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:38.045038 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:38.091066 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:15:38.131906 14226 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:38.132084 14226 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:15:38.132678 14226 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:15:38.133657 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:38.133759 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:15:38.133843 31125 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
W20240501 14:15:38.133960 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:38.133980 14226 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:38.135854 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:38.186973 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:38.192497 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:38.216097 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:38.254616 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:38.282608 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:38.370184 14226 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:38.450654 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:38.476563 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 14:15:38.478695 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:38.589267 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:38.654186 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:38.728817 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:38.733861 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:38.762159 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:38.764729 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 14:15:38.788877 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:38.809152 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:38.843662 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:38.912248 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:38.942862 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:15:38.995438 14558 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:39.006059 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:39.018429 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:39.049258 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:39.088598 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:39.105485 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:39.204512 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 14:15:39.277541 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:39.414463 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:39.422744 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:39.455403 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:39.465999 14226 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:39.534129 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:39.542141 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:39.571404 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:39.751410 31386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 75 similar messages]
W20240501 14:15:39.891552 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:15:39.916352 14226 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:39.954658 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:39.956220 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:39.966909 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 14:15:40.005844 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 14:15:40.023236 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:40.028458 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:40.030459 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:40.039137 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:15:40.111516 14558 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:40.111663 14558 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:40.112145 14558 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:15:40.112566 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:15:40.112752 31581 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:15:40.113299 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:40.113390 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:15:40.113451 31125 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:15:40.113569 14558 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:40.122576 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:40.142721 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:15:40.167122 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 14:15:40.211032 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:40.234578 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:15:40.348444 14558 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:40.418320 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:40.461648 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:40.462021 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:40.477835 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:40.497646 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:40.504875 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:40.558172 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:40.561964 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:40.600590 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:40.619673 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:40.653410 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:40.662174 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:40.694526 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:40.735253 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:40.736543 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:40.760495 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:15:40.794071 14635 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:40.856287 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:40.953933 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:41.046854 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 14:15:41.070583 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:41.090054 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:41.110406 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:41.132180 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:41.137495 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:15:41.258332 14558 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:41.283574 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 14:15:41.336637 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:41.356549 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:41.365692 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:41.438058 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:41.442080 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:41.491182 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:41.559083 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:41.562613 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:41.618151 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:41.656369 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:41.697667 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:41.804536 14763 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:41.829005 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:41.853534 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:41.853658 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 14:15:41.924918 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 14:15:41.958554 31123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 14:15:41.978737 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:41.994510 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:42.055227 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:42.101701 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:42.149631 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:42.165051 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:15:42.330902 14763 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:42.344172 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 14:15:42.448277 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:42.522050 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:42.572126 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:42.572216 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:42.598987 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 14:15:42.639708 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:42.643678 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:42.665901 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:42.702647 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:42.751739 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:42.808656 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:42.826709 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:15:42.870990 14558 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:42.959271 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:42.973678 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:15:43.026923 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.055150 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.055493 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 14:15:43.075753 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 14:15:43.101825 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.106458 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.140892 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.146008 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:43.151670 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:43.164263 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.171767 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.172672 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.182122 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:43.197881 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.316793 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240501 14:15:43.345252 14558 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:43.346794 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.392094 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:43.530900 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.614457 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.689051 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.695086 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:43.699005 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:43.710291 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:43.716797 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.749941 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240501 14:15:43.789649 14558 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:43.900789 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:43.980896 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:43.988662 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:44.047657 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:44.047950 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:44.062613 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:44.107461 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:44.206776 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:15:44.213738 14763 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:44.215308 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:44.236352 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:44.309638 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:44.345146 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:44.345894 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:44.368522 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 14:15:44.617435 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:44.672657 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:44.719213 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:44.724086 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:44.775842 14986 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:44.782513 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:44.817780 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:15:44.824636 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:44.850754 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:44.860596 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 14:15:44.927009 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:44.956074 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:15:44.994391 14986 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:44.994539 14986 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:44.995090 14986 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:15:44.995391 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:15:44.995559 31581 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:15:44.996279 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:15:44.996369 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:44.996452 31124 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:15:44.996593 14986 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:45.076046 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:45.111063 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:45.175436 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:45.193449 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:45.200321 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:45.277870 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:45.365487 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:45.365490 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240501 14:15:45.369722 15034 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:45.376962 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:45.388715 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:45.464524 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:45.473819 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:45.480170 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:15:45.517225 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 14:15:45.530527 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:45.548143 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:45.570544 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:45.605254 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:45.620374 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:45.642105 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:45.717854 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:45.724673 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:45.734524 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:45.779052 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 14:15:45.840176 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:45.914763 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:45.954983 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:15:45.991636 15034 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:46.065042 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 14:15:46.118907 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:46.206049 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:46.218330 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:46.270728 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:46.336455 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:46.354414 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:46.410145 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:46.412237 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:46.415433 14986 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:46.439379 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:46.557359 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:46.610780 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:46.618714 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:46.655441 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:46.713320 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:46.765779 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:46.791450 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:46.801815 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:46.812036 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:46.828727 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:15:46.886233 15034 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:46.929540 31254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 14:15:46.949589 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 14:15:46.970026 31123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240501 14:15:47.070870 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:47.103456 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:47.114362 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:47.193312 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:47.328069 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:47.338872 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:47.371127 14986 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:47.397442 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:47.408142 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:47.414980 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 14:15:47.523531 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:47.534968 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:47.543370 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:47.545436 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:47.607106 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:47.610095 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:47.650672 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:47.697826 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 14:15:47.832615 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:47.839936 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:15:47.845635 15034 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:47.856416 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:47.894547 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:15:47.927022 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:48.003439 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:48.043061 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:48.054649 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:48.062090 31516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 14:15:48.062700 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:48.135526 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:48.138609 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 14:15:48.148972 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:48.153501 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:48.153630 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:48.156919 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:48.226464 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 14:15:48.249464 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:48.328087 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:48.387180 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:48.405933 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 14:15:48.441170 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:48.441859 15034 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:48.541051 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:48.628561 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:48.676978 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:48.684128 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:48.786020 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:48.814590 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:48.849139 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:48.933494 15271 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:48.980719 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:49.021845 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:49.066358 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:49.087461 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:49.103056 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:49.120644 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:49.211125 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:49.211655 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 14:15:49.288923 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:49.372046 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:49.406517 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:49.457896 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:15:49.511938 15348 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:49.556419 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:49.569746 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:49.616658 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:49.695220 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:49.701503 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:49.754662 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 14:15:49.827903 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:49.842854 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 14:15:49.843469 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:49.931416 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:49.969154 15348 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:49.971525 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:50.027555 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:50.033897 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:50.086864 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:50.098961 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 14:15:50.141275 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:50.141790 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:50.184707 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:50.318796 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:50.321367 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:50.335430 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:50.360417 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:50.394007 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:50.409430 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 14:15:50.421967 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:15:50.473052 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:15:50.507344 15417 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:50.539034 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:50.549336 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:50.565725 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:50.587026 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:50.599148 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:50.630097 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:50.658248 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:50.679405 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:50.717146 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 14:15:50.728523 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:50.749063 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:50.880991 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:50.942121 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240501 14:15:51.037288 15417 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:51.054533 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:51.091089 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:51.106163 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:51.157176 15417 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:51.157301 15417 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:15:51.158064 15417 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:15:51.158619 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:15:51.158753 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:51.158810 31124 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
W20240501 14:15:51.158922 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:15:51.159106 15417 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:51.160576 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:51.163194 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:51.228320 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:51.295830 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:51.370708 31032 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 7a542c07a3f841b0a31051a8a37083bf with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
W20240501 14:15:51.381815 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:51.490137 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:51.515362 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:51.537566 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:15:51.616451 15122 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:15:51.616582 15122 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:51.617168 15122 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:15:51.618117 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:15:51.618254 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:51.618335 31517 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
I20240501 14:15:51.618477 15122 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:51.618635 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:15:51.631325 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:51.639267 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:51.648037 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240501 14:15:51.648049 15509 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:51.666136 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:51.719605 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:51.856086 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:51.916692 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:51.929338 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:51.932499 31254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 14:15:51.973338 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:51.988540 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 14:15:52.096619 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:52.120874 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:15:52.167055 15417 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:52.195123 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:52.208626 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:52.238798 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:52.246918 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:52.252292 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:52.330140 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 14:15:52.437355 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:52.489099 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:52.491354 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:52.492874 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:52.607774 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:52.617763 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:52.721226 15417 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:52.725620 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:52.792187 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:52.823884 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 14:15:52.863436 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:52.864939 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:52.891481 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:52.949898 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 14:15:52.955152 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:52.956081 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:53.003890 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:53.025552 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:53.036315 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:53.068212 31034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240501 14:15:53.068987 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b1006490ec2841b2a3365295caaf19e1: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:15:53.094383 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:15:53.101220 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240501 14:15:53.103071 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:53.130072 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:53.169723 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:53.191501 15417 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:53.196671 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:53.243515 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:53.245786 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 14:15:53.262502 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:53.313971 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:53.435109 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:53.464301 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:53.469363 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:53.472790 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:53.494976 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:15:53.630407 15417 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:53.647907 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:53.656340 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:53.662020 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 14:15:53.760962 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:53.766142 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:53.945504 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:54.029460 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:54.055344 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:54.067615 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:54.120196 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 14:15:54.144181 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:15:54.236088 15417 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:54.249351 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:54.261286 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:54.265187 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:54.301163 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:54.341282 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:54.372659 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:54.417577 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6f8bd03c05d84fdaaf138a6d0608e30e: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:15:54.569559 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:54.658156 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:54.670658 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:54.681005 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:54.768371 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:54.770325 15735 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:54.803319 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:54.812520 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 14:15:54.845901 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:15:54.856600 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:54.872534 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:54.874912 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:54.895912 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:55.071856 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:55.086405 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:55.189182 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 14:15:55.194465 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:55.219851 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:55.224375 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:55.242766 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:55.250128 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:55.285591 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:55.347908 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:15:55.380597 15735 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:55.470088 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 14:15:55.483888 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:55.492723 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 14:15:55.540941 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:55.605819 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:55.645471 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:55.647177 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:55.659859 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:15:55.666558 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:55.682521 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:55.703594 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:55.784935 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:55.828294 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:55.884162 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:55.925129 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:55.926496 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:15:55.949859 15828 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:56.024540 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:56.135236 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 14:15:56.142055 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:56.169237 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:56.199729 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:56.226601 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:56.269553 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:56.401448 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:56.424329 15735 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:56.442414 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 14:15:56.451598 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:56.528252 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:56.550050 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:56.556104 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:56.578418 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:56.627785 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:56.688524 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:15:56.808162 15828 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:56.820674 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:56.899019 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:56.970140 31254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 14:15:56.971422 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:56.977069 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:56.990948 31124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 14:15:56.991516 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:57.029490 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:57.065101 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:57.122381 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:57.154856 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:57.202214 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:57.243160 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:15:57.288117 15735 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:57.323036 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 14:15:57.397701 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:57.398337 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:57.532677 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:57.534298 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:57.545670 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:57.550992 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:57.581689 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:57.593649 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:57.597806 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:57.607496 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 14:15:57.639374 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:57.709882 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:57.819083 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:57.859639 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:15:57.871611 15967 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:57.886826 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:57.926126 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 14:15:57.966709 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:58.014282 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:58.015573 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:58.081205 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:58.102121 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240501 14:15:58.102770 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:58.153986 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:58.165349 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:58.188377 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:15:58.205653 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:15:58.250775 15967 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:15:58.250902 15967 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:15:58.251315 15967 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:15:58.251688 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:15:58.251850 31581 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:15:58.252223 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:58.252328 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:15:58.252382 31125 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:15:58.252503 15967 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:15:58.258921 15735 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:58.277246 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:58.336959 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:58.348202 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 14:15:58.357350 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:58.414695 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 14:15:58.486045 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:58.501812 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:58.516300 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:58.548998 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:58.607972 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:58.697636 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:58.808203 15735 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:58.904384 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:58.908051 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:58.956207 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:58.968966 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:15:58.990658 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:58.996457 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:59.039499 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 14:15:59.048076 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:59.070169 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:59.073189 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:59.149132 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:15:59.197618 15735 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:59.214226 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:59.417729 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:59.423346 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:59.432893 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:59.557477 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:59.608775 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:15:59.637530 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:15:59.659744 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:15:59.702970 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:15:59.718248 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:15:59.748759 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:15:59.764227 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:15:59.813467 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:15:59.813531 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:59.893648 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:15:59.897095 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:15:59.915405 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:15:59.959756 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 14:15:59.975445 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:15:59.984855 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:16:00.033969 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:00.092125 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:00.176688 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 14:16:00.180240 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:16:00.231467 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:00.288820 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:00.348287 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:00.352921 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:00.365430 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:00.384855 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:00.405932 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:00.465970 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:00.475319 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:00.507485 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:00.632411 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 14:16:00.634917 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:00.662904 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:00.738135 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:00.745980 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:00.748656 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240501 14:16:00.780403 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:00.799088 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:00.827507 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:16:00.916648 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:00.919735 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 14:16:00.924562 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:01.010676 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:16:01.011461 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:01.040195 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:01.059039 15983 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:01.059197 15983 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:16:01.059689 15983 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:16:01.060742 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:16:01.060869 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:01.060961 31517 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
W20240501 14:16:01.061043 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:16:01.061108 15983 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:01.134294 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:01.168598 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:01.204627 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:01.228932 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:01.232721 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:01.253067 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:16:01.260668 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:16:01.328861 16036 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:16:01.328999 16036 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:16:01.329561 16036 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:16:01.330072 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:16:01.330215 31581 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:16:01.330632 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:16:01.331102 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:01.331161 31124 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:16:01.331282 16036 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:01.346814 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:01.375944 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:01.466028 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 14:16:01.509358 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:01.529486 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:01.627542 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:01.648216 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:16:01.665256 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:01.714736 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:01.724537 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:01.782845 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:01.793181 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:01.961807 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:01.972954 31254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 14:16:02.018548 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 14:16:02.058385 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:02.078734 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:02.098389 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:02.148519 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:02.170490 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:02.170631 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:02.171324 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:16:02.187077 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:02.192878 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:02.260983 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:02.287017 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 14:16:02.365871 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:02.402269 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 14:16:02.447175 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:02.457965 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:02.474843 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:02.509985 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:02.633188 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:02.662684 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:02.664809 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:16:02.672186 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:02.688612 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:02.736330 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:02.794456 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:02.801290 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:02.857342 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:02.940223 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:02.967068 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:02.967446 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:03.056142 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:03.056339 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:03.092938 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 14:16:03.142506 31516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240501 14:16:03.160526 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:03.215061 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 14:16:03.235476 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:03.277349 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:03.350713 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:03.363780 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:03.380224 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:03.407873 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:03.411254 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240501 14:16:03.466389 16284 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:03.564308 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:03.585454 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:16:03.646567 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:03.684345 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:03.706331 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:03.760344 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:03.854267 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:03.888427 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 14:16:03.891415 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:03.908892 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:16:03.948156 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:03.967464 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:04.023478 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:04.025249 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:04.133176 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:04.187758 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:04.223544 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:04.241586 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:04.242498 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:04.243140 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:04.252056 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:04.390839 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:04.393492 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:16:04.406832 16284 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:04.525149 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:04.583214 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 14:16:04.594349 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:04.598136 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:04.637215 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:04.665488 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:04.738595 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:04.764863 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:04.775477 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:16:04.878365 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:04.938875 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:04.942262 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 14:16:04.944358 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:16:04.967730 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:04.995685 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:05.066428 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:05.154388 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:05.275588 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:05.286434 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:05.336427 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:05.350688 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:05.375123 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:05.385771 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:16:05.397390 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:05.399149 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:05.520089 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:05.520454 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 14:16:05.522478 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:05.530359 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:05.603523 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 14:16:05.624967 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:05.656266 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:05.665138 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:05.684036 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 14:16:05.756917 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:05.777554 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:05.791832 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:05.868491 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:05.894474 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:16:05.910465 16562 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:05.941831 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:05.948011 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:06.088523 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:06.130446 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 14:16:06.177482 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:16:06.207372 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:06.248116 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:06.249045 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:06.253625 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:06.475651 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:06.526239 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:06.532079 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:06.549932 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:06.552191 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:06.691368 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:06.695324 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:06.709040 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:06.756047 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:06.776943 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:06.820843 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:06.831369 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:06.845614 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:06.910389 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:06.935070 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:06.939668 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:07.026854 31255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 14:16:07.051995 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:07.052284 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 133 similar messages]
I20240501 14:16:07.059682 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:07.083588 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:07.092542 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:07.195307 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:07.281697 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:07.287314 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:07.354974 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 14:16:07.356618 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 14:16:07.360648 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:07.539584 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:07.540939 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:07.617061 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:07.665561 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:07.676543 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:07.680032 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:07.744879 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:07.745659 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:07.767356 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:07.828425 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:07.864739 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:07.867889 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:07.885990 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 14:16:07.892438 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:07.914130 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 14:16:07.976946 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:07.994038 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:07.998091 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:08.011374 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:08.087841 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:08.135519 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:08.137483 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:08.201375 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 14:16:08.210657 31516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 14:16:08.240979 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:08.243870 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:08.287767 16036 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:08.315340 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:08.416114 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:08.482530 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:16:08.512086 16362 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:16:08.512250 16362 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:16:08.512822 16362 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:16:08.513345 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:16:08.513531 31187 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:16:08.513931 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:16:08.514040 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:08.514102 31517 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:16:08.514225 16362 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:08.555704 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:08.645893 16036 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:16:08.646044 16036 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:16:08.646498 16036 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:16:08.647465 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:16:08.647663 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:08.647751 31124 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
W20240501 14:16:08.647852 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:08.647969 16036 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:16:08.682631 16686 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:08.731751 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 14:16:08.746570 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:08.750291 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:16:08.760121 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:08.805383 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:08.817464 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:08.961922 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:09.084793 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:09.095842 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:16:09.157785 16686 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:09.179138 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:09.223424 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:09.244246 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:09.247000 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:09.250291 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:09.301812 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:09.310789 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:09.333118 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:09.348791 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:09.431730 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:16:09.555215 16686 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:09.574365 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:09.721101 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:09.743068 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:09.746783 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 14:16:09.770601 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:09.773546 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:09.783545 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:09.811020 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:09.831954 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:09.836593 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:09.862671 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:09.953058 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 77 similar messages]
I20240501 14:16:09.954007 16686 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:09.970665 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:10.060227 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 14:16:10.088058 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:10.089365 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:10.101270 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:10.111069 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:10.131387 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:10.137663 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:10.149622 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:10.262826 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:10.314878 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:10.350329 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 14:16:10.539008 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:10.544979 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:10.547593 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:10.551962 16686 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:10.560549 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:10.568770 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 14:16:10.613715 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:10.623275 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:10.649298 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:10.668408 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:10.702682 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:10.733014 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:10.790760 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:10.843117 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:10.957644 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240501 14:16:10.990895 16036 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:16:10.991047 16036 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:16:10.991514 16036 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:16:10.992707 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:10.992801 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:16:10.992882 31125 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
W20240501 14:16:10.992991 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:10.993033 16036 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:11.022550 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:11.092357 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:11.102020 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:16:11.148291 16686 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:11.171075 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:11.249094 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 14:16:11.299211 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:11.361403 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:11.453786 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:11.468329 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:11.514531 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:16:11.561218 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:11.604614 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:11.616633 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:11.659564 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:11.660133 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:11.746016 16686 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:11.840404 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:11.862068 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:11.907583 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:11.938588 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:12.023161 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:12.042523 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:12.057897 31123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240501 14:16:12.060344 31254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 14:16:12.061081 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:12.137487 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:12.159463 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:12.224433 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:12.258463 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:16:12.299510 16686 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:12.313004 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:12.367636 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:12.378926 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:12.397388 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 14:16:12.418341 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:12.419376 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:12.427047 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:12.501542 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 14:16:12.506906 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:12.552173 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:12.561281 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:12.621970 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:12.623059 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:12.652688 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:12.721890 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:12.745205 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 14:16:12.763003 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:12.780534 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:12.815832 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:12.826925 16686 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:12.893448 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:13.001084 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:13.064105 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:13.091287 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 14:16:13.142122 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:13.168394 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:13.172911 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:13.195737 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:13.258423 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:13.271461 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 14:16:13.281122 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 93 similar messages]
I20240501 14:16:13.293687 17075 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:13.376616 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:13.501634 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:13.533571 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:13.605235 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:13.650039 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:13.660133 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:13.684141 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:16:13.732770 17075 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:13.771576 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:16:13.785778 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 14:16:13.789887 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:13.856977 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:13.897917 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:13.979635 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:13.983919 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:16:14.104542 17075 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:16:14.104676 17075 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:16:14.105139 17075 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:16:14.106428 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:16:14.106575 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:14.106660 31123 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a0caf2974041408bb32362bfc4284eaa, a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:16:14.106770 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:14.106791 17075 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:14.200620 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:14.214816 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:14.232266 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:14.247311 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:16:14.267752 17075 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:14.279707 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:14.417786 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:14.426999 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:14.437287 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:14.542662 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:14.656502 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 14:16:14.666934 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:14.674634 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:14.693279 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 14:16:14.711269 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:14.714902 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:14.718297 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:14.759101 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:14.767467 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:14.801070 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:16:14.812752 17075 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:14.844987 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:14.912068 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:14.955557 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:14.973457 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:15.014362 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:16:15.038086 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:15.069439 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:15.123160 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 14:16:15.211104 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:15.216184 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:15.230717 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:15.240725 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:15.241318 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:15.314218 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:15.315337 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:15.361825 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 14:16:15.369421 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:15.371843 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:16:15.395011 17243 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:15.395133 17243 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:16:15.395565 17243 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:16:15.397284 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:16:15.397411 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:16:15.397466 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:16:15.398133 17243 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:15.398452 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:15.421355 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:16:15.427254 17075 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:15.516774 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:15.526309 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:15.587579 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:15.706290 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 14:16:15.799001 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:15.873731 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:15.889681 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:15.917786 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:15.975522 17075 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:16.013247 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:16.031858 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:16.070114 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:16.161345 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:16.199892 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:16.258569 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:16.278788 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:16.326195 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:16:16.454851 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 14:16:16.476982 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:16.489663 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:16.493990 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:16.584071 17317 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:16.596511 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:16.687206 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:16.705349 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:16.717175 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:16.817365 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:16.895754 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:16.946410 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:16.968160 17317 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:16.973526 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:16.977288 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:17.002257 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:17.004226 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:17.053543 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:17.090986 31123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240501 14:16:17.099891 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:17.110441 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:17.112448 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 14:16:17.113137 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 14:16:17.113656 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:17.172209 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 14:16:17.193509 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:17.214290 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:17.300617 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:17.345497 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:17.402325 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:17.474617 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:17.494694 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:17.507262 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:17.509629 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:17.525969 17317 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:17.633263 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:17.646250 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:17.684725 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:17.696035 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:17.723675 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:17.752097 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:17.797865 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:17.851398 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 14:16:17.899071 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:17.929591 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 14:16:17.950893 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:17.962584 17438 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:17.982822 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:18.011297 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:18.027974 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:18.236536 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:18.300416 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:18.314744 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 14:16:18.391355 31516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240501 14:16:18.391988 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:18.408696 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:16:18.462673 17438 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:18.479333 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:18.532394 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:18.631806 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:18.675503 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:18.690160 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:18.722148 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:18.758037 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:16:18.801820 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:18.811671 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:16:18.920188 17438 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:18.926319 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:19.040755 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 14:16:19.086200 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:19.187600 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:19.195143 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:19.223313 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:19.281935 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:16:19.314788 17317 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:19.316262 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:19.367599 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:19.444602 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:19.504913 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:19.544116 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:19.554134 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:19.570972 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:19.586817 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:19.591077 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:19.593253 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:19.596177 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:19.598613 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 14:16:19.666158 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:19.677199 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 14:16:19.699587 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:19.733371 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:16:19.763767 17317 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:19.831058 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:19.846247 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:19.870395 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:19.920121 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:19.928354 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:16:19.968228 17523 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:16:19.968380 17523 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:16:19.968945 17523 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:16:19.969372 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:16:19.969543 31187 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:16:19.970100 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:16:19.970230 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:19.970290 31517 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:16:19.970412 17523 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:20.045497 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:16:20.097548 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:20.149907 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:20.162094 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 14:16:20.165011 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:20.181805 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:20.215392 17438 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:20.230361 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 14:16:20.280267 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:20.349960 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:20.371740 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:20.423702 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:20.425483 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:20.487409 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:20.499765 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:20.533895 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:20.598423 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:20.601958 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:20.615283 17317 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:20.741268 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 14:16:20.770351 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:20.856535 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:20.888052 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:20.922170 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:20.977870 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:21.020220 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:16:21.142201 17438 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:21.162698 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:21.203807 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:16:21.230804 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:21.285174 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:21.295876 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:21.356976 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:21.424008 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:21.472884 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:21.565115 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240501 14:16:21.598316 17438 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:21.613867 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:21.740064 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:21.759944 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:21.872392 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:21.888562 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:21.924621 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:21.932516 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:21.956672 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:22.006001 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:22.027695 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:22.091954 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:22.122088 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 14:16:22.122694 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 14:16:22.128046 31254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 14:16:22.128654 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:22.155076 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:22.183727 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:22.184513 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:22.197818 17783 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:22.202816 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:22.217841 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:22.222863 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:22.229120 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:22.247303 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 14:16:22.277667 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:22.329695 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:22.437335 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:22.462592 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:22.572284 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:22.577142 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 14:16:22.593554 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:22.612159 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:22.627435 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:16:22.644937 17438 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:22.681929 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 14:16:22.751888 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:22.881029 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:22.886456 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:22.918881 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:22.959815 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:22.961380 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:22.961433 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:23.052700 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:23.063946 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:23.087334 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:23.129325 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:16:23.139750 17783 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:23.143220 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:23.296038 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 14:16:23.311671 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:23.334589 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:23.364318 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:23.455499 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 14:16:23.457168 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:23.564369 17438 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:23.636358 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:23.648948 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:23.698017 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:23.725782 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:23.746610 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:23.787117 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:23.792788 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:23.863440 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:23.868197 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:16:23.928431 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:24.027745 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 14:16:24.044624 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:24.090458 17920 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:24.139775 31032 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 7a542c07a3f841b0a31051a8a37083bf with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
W20240501 14:16:24.158140 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:24.231796 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:24.255616 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:24.343894 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:24.345683 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:24.365018 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:24.408232 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:24.516113 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:24.529209 17920 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:24.582608 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:24.695410 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:24.708447 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:24.710685 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:24.738487 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:24.739121 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 14:16:24.794955 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:24.826805 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:24.839943 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:24.871412 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:24.892657 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:24.924032 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:24.932196 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 14:16:24.943109 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:24.998374 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:25.080914 31386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:16:25.081553 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:25.085739 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:25.111866 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:25.118956 17999 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:25.174782 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:25.195227 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:25.203373 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 14:16:25.279939 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:25.296227 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 14:16:25.350337 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:25.360710 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:25.394693 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:25.453321 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:25.459913 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:25.500905 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:25.647531 17999 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:25.677855 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:25.698879 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:25.709774 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:25.769929 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:25.826187 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:25.855913 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:25.896957 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:25.948565 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:25.957499 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:25.990047 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 14:16:26.108843 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:16:26.122248 17920 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:26.139175 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:16:26.245012 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:26.251080 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:26.256438 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:16:26.300645 18065 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:26.300786 18065 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:16:26.301285 18065 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:16:26.302245 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:26.302353 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:16:26.302438 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
I20240501 14:16:26.302561 18065 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:26.302572 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:16:26.319554 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:26.346439 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:26.346508 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:26.359021 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:26.417245 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:26.506577 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:26.544407 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 14:16:26.551676 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:26.609895 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:26.687407 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:26.735774 18129 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:26.838855 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:26.854998 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:26.870651 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:26.906038 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:26.960120 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:27.077852 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:27.103498 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:16:27.158622 17920 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:27.163189 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 14:16:27.193063 31255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 14:16:27.238488 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:27.263298 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:27.264083 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 14:16:27.300642 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:27.357247 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:27.364790 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 14:16:27.373368 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:27.396003 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:27.408329 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:27.429047 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:27.479401 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:27.489042 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 14:16:27.510284 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:27.525417 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:27.550268 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:27.645787 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:27.656672 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:27.670439 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240501 14:16:27.692474 18199 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:27.726783 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:27.778209 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:27.875911 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:27.894196 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:27.899463 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:27.929375 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:27.969153 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:27.985999 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:28.018812 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:28.041575 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:28.048852 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:28.072238 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:28.108870 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:16:28.236469 18199 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:28.250934 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:28.297549 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:28.410744 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:28.443823 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:28.469415 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 14:16:28.564692 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:28.577997 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240501 14:16:28.630120 18199 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:16:28.644496 18199 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:16:28.644646 18199 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:16:28.645216 18199 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:16:28.645689 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:16:28.645859 31580 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:16:28.646298 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:16:28.646392 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:28.646456 31124 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:16:28.646606 18199 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:28.647578 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:28.686444 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:28.689637 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:16:28.792778 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:28.796080 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:16:28.869498 18199 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:16:28.869638 18199 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:16:28.870121 18199 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:16:28.871029 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:16:28.871395 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:28.871465 31124 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
I20240501 14:16:28.871590 18199 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:28.871621 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:28.899955 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:28.919895 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:28.961853 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:28.991426 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:29.055578 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240501 14:16:29.111845 18199 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:29.129884 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:29.133896 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:29.147739 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:29.180989 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:29.264600 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:29.268007 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:29.348824 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:29.382615 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:29.555593 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:29.556367 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:29.587914 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:29.590106 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:16:29.607499 17920 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:29.633199 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:29.774695 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 14:16:29.795130 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:29.842743 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:29.869678 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:29.900662 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:29.903244 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 14:16:29.906558 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:29.932937 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:29.944258 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:30.000564 17920 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:30.003194 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:30.034052 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:30.076153 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 14:16:30.113173 31386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:16:30.122944 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:30.126072 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:30.144443 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:30.152448 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:30.152535 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:30.168881 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:30.300680 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:30.325477 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 14:16:30.350129 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:30.371701 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:30.383142 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:30.384888 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:16:30.389952 17920 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:30.391177 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:30.398180 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:30.489627 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:30.530411 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:30.704350 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:30.767481 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:30.810267 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 14:16:30.859403 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:30.931926 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:30.989262 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:30.991101 18199 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:31.188529 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:31.193028 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:31.207545 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:31.259943 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:31.263478 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:16:31.391472 18199 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:31.415236 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:31.417212 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:16:31.422734 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:31.457799 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:31.461290 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:31.507150 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:31.531996 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:31.681752 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:31.762995 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 14:16:31.766126 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:31.780804 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:31.869081 17920 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:31.897157 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:31.982929 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:31.997368 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:32.000818 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:16:32.009438 17920 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:16:32.009595 17920 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:16:32.010102 17920 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:16:32.010993 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:32.011240 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:32.011343 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:16:32.011428 31125 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
W20240501 14:16:32.011516 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:32.011590 17920 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:32.066804 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 14:16:32.094588 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:32.099819 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:32.115036 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:32.192687 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:16:32.208905 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 14:16:32.235574 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:16:32.294081 18199 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:32.323282 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:32.394114 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:32.461633 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:32.466816 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:32.467343 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:32.479398 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:32.495976 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:32.504501 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:32.595834 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:32.669783 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:16:32.679687 17920 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:32.682188 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:32.697190 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:32.707698 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 14:16:32.711239 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 14:16:32.720170 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 14:16:32.762200 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:32.804085 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:32.869295 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:32.876412 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:16:32.898226 18340 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:32.898361 18340 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:16:32.898789 18340 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:16:32.899889 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:32.899995 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:16:32.900055 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
W20240501 14:16:32.900202 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:16:32.900206 18340 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:32.923779 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:32.937341 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:32.956084 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:33.006685 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:33.021976 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:33.026033 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:33.092193 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:33.151504 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 14:16:33.186573 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:33.203963 18609 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:33.338650 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:33.395494 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:33.396988 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:33.446285 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:33.501919 31516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 14:16:33.502594 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:33.630823 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:33.723059 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:33.738636 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:33.755359 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:33.755916 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:33.803337 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:33.835404 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:33.997290 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:34.037529 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:34.077713 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:16:34.085134 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:34.126207 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:34.226847 18688 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:34.234172 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:34.235349 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:34.279181 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 14:16:34.336179 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:34.486675 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:34.510943 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:34.552479 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:34.599926 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:34.604290 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:34.611496 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:34.634073 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:16:34.652535 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:34.724653 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:34.738461 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:34.766593 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 14:16:34.792850 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:34.822012 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:34.929821 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:34.963449 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:34.994793 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:35.012326 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:35.079382 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:35.081676 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:35.093529 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:35.108341 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:35.111085 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:35.137549 31386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:16:35.173039 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:35.199828 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:35.244511 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:35.281389 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 14:16:35.286358 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:35.315575 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 14:16:35.317817 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 14:16:35.322479 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:35.340797 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:35.368901 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:35.371060 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:35.396178 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:35.423297 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:35.491168 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:35.538760 18688 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:35.577898 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:35.605930 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 14:16:35.638502 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:35.717940 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:35.734694 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:35.880769 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:35.889464 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:35.900372 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:36.017100 18688 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:36.043946 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:36.146415 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:36.344880 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:36.356344 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:36.359493 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:36.374423 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:36.421078 18688 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:36.439255 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:16:36.471356 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:36.496318 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:36.566888 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:36.575356 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:36.601362 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:36.760915 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:36.810042 18688 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:36.819119 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:36.824777 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 14:16:36.848243 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:36.849200 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:36.897394 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:36.960973 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:36.994269 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:37.038115 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:37.078995 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:37.134177 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:37.148710 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:37.185705 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 14:16:37.217574 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 14:16:37.220957 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 125 similar messages]
W20240501 14:16:37.267490 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:16:37.331234 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:37.360344 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:37.384564 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:37.459569 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:37.488536 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:37.494407 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:37.517819 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:37.538645 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:37.541877 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:37.560215 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:37.571291 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:37.574842 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:37.630578 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 14:16:37.740309 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:37.747467 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:37.807019 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:37.858100 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:37.869040 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:37.871286 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:37.891590 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:37.924281 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:37.939704 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 14:16:37.967474 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:37.981990 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 14:16:37.991906 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:38.020341 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:38.078320 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:38.125978 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:38.185868 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 14:16:38.311354 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:38.383409 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:38.406309 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:38.469025 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:38.513980 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 14:16:38.514606 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:38.521915 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:38.582489 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:38.794450 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:16:38.803879 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:38.835501 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:38.835611 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:38.875442 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:38.879799 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:38.965724 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:39.030691 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:39.053303 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:39.080217 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:39.115986 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:39.205921 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:39.214011 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:39.227370 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:39.277726 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 14:16:39.284462 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:16:39.288851 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:39.309499 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:39.420639 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:39.523288 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:39.621713 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:39.642210 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:39.694905 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:39.698652 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:39.702795 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:39.742182 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:16:39.828518 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:39.852486 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:39.917645 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:39.932720 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 14:16:39.945142 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:16:39.955219 18655 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:16:39.955375 18655 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:16:39.955828 18655 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:16:39.956212 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:16:39.956367 31580 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:16:39.957218 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:39.957425 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:16:39.957506 31125 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:16:39.957643 18655 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:39.963840 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:39.981612 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 14:16:39.986104 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:39.999465 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:40.015385 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:40.019131 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:40.040997 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:40.057040 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:40.137359 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:40.144922 31386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:16:40.248129 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:40.269529 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:40.305502 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:40.321220 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:40.336885 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:40.362627 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:40.398237 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:40.410133 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:40.420213 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:40.421170 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240501 14:16:40.428095 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:40.452169 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:40.464681 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:40.477973 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:40.602283 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 14:16:40.805514 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:40.849192 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:40.877509 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240501 14:16:40.891112 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:40.907536 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:40.964757 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:41.079710 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:41.176522 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:16:41.276746 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:16:41.297652 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:41.305950 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:41.353927 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:41.384594 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:41.435025 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:41.484807 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:41.506812 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:41.537787 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:41.570492 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:41.623462 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:41.632058 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:41.754063 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:16:41.769659 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:41.794991 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:41.850831 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:41.880910 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:41.928368 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:42.023167 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:42.040601 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:42.106627 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:42.209371 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 14:16:42.222641 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 130 similar messages]
I20240501 14:16:42.240109 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:42.242025 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:42.255472 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 14:16:42.256275 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 14:16:42.316241 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:42.320678 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:42.356379 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:42.415377 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:42.421872 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:42.422021 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:42.442190 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:42.463169 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 14:16:42.484586 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:42.502830 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:42.535856 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:42.612926 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:42.625907 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:42.627817 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240501 14:16:42.630007 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:42.648667 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:42.684417 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:42.688026 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:42.746538 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:42.752807 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:42.801481 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:42.811152 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:42.812104 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:42.850365 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:42.852171 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:42.877331 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:42.929600 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:43.032503 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:16:43.056676 19156 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:43.081032 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:43.139706 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:43.218676 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 14:16:43.225638 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:43.240674 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 14:16:43.391916 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:43.402745 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:43.468238 19156 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:43.583861 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:43.601740 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:43.635288 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 14:16:43.703822 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:43.747586 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:43.774497 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:16:43.775832 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:43.827680 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:16:43.896776 19156 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:43.905584 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:43.965345 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:43.980111 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:44.030716 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:44.107499 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:44.109419 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:44.155663 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:44.206152 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:44.236001 18655 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:16:44.236140 18655 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:16:44.236629 18655 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:16:44.237871 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:44.237946 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:16:44.237987 31124 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
I20240501 14:16:44.238171 18655 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:44.238178 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:44.361332 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:44.378286 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:16:44.407970 19156 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:44.435390 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:44.453049 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:44.495586 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:44.537304 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:44.597551 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 14:16:44.643890 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:44.683735 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 14:16:44.692000 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:44.716079 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:44.770515 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:16:44.844563 19156 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:44.857208 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:44.932339 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 14:16:44.936669 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:44.941661 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:44.958842 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:44.973444 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:45.008512 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:45.018384 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:45.027309 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:45.028569 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:45.031035 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:45.097930 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:45.121161 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:45.135079 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 14:16:45.156461 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:45.201253 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:16:45.230578 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:45.265581 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:45.298404 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:45.347903 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:16:45.386875 19156 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:45.390347 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:45.395397 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:45.411003 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:45.476737 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 14:16:45.476863 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:45.477561 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:45.477921 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:45.524230 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 14:16:45.561530 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:45.801105 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:45.804419 18655 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:16:45.853112 19541 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:45.853264 19541 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:16:45.853750 19541 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:16:45.854821 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:16:45.855086 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:16:45.855151 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
I20240501 14:16:45.855350 19541 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:45.855535 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:45.871723 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:45.874346 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:45.928098 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:46.030335 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:46.228940 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:46.255178 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:46.260697 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:46.320504 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:46.341164 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:16:46.365694 19605 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:46.420408 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:46.435509 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:46.455816 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:16:46.490062 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:46.571805 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:46.614375 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:46.642678 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:46.646466 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:46.784538 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:46.857249 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:16:46.857435 19605 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:46.895078 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:46.904317 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:46.969437 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 14:16:47.046846 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:47.115155 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:47.152107 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:47.179471 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:47.220919 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:47.246660 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:47.255705 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240501 14:16:47.258462 31255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 14:16:47.302004 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:16:47.330791 19605 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:47.332247 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:47.346607 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:47.355604 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:47.412750 31036 maintenance_manager.cc:419] P d913062e66274883a568f59820e333ee: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033386
I20240501 14:16:47.415580 31035 maintenance_manager.cc:643] P d913062e66274883a568f59820e333ee: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.001s sys 0.000s Metrics: {"bytes_written":1751,"delete_count":0,"lbm_write_time_us":128,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":74,"threads_started":1,"update_count":19,"wal-append.queue_time_us":155}
I20240501 14:16:47.415944 31036 maintenance_manager.cc:419] P d913062e66274883a568f59820e333ee: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.114933
W20240501 14:16:47.421433 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:16:47.422683 31035 maintenance_manager.cc:643] P d913062e66274883a568f59820e333ee: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.002s sys 0.004s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":15121,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":274,"lbm_read_time_us":186,"lbm_reads_lt_1ms":17,"lbm_write_time_us":241,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":112,"threads_started":2,"update_count":19}
W20240501 14:16:47.451313 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:47.461297 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:47.481179 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 14:16:47.515902 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:47.581068 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:47.623661 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:47.641252 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:47.656143 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 14:16:47.666900 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:47.699932 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:47.704018 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:47.712003 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:47.736141 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 14:16:47.740358 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 14:16:47.794026 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:16:47.836562 19605 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:47.877880 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:47.878902 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:47.956465 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:47.970127 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:47.984685 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:47.989357 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:16:48.008759 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 14:16:48.070062 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:48.208523 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:48.245117 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:48.274248 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:48.321841 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:16:48.342087 19763 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:48.422032 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:48.516353 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:48.564349 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:48.623019 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:48.671152 31516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 14:16:48.671784 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:48.775810 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:48.798888 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:48.929363 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:16:48.939106 19763 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:48.960291 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:48.996166 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:49.082026 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:49.098613 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:49.144469 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:49.149948 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:49.168543 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:49.179584 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:49.186110 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:49.209154 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:49.394726 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:49.411512 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:49.417109 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240501 14:16:49.430063 19797 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:49.461805 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:49.563366 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:49.573923 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:49.582414 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:49.582968 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:49.597887 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:49.668926 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:49.673130 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:49.768985 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:49.779474 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:49.797639 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:49.874140 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 14:16:49.944712 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:49.969609 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:16:50.028605 19891 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:50.041716 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:50.042863 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:50.054807 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 14:16:50.080056 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:50.110194 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:50.110957 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:50.114511 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:50.119014 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:50.134107 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:50.169628 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 14:16:50.222846 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 14:16:50.251367 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:50.286209 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 14:16:50.301899 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 14:16:50.411770 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:16:50.437649 19797 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:50.490010 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:50.512416 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:50.522080 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:50.538000 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:50.626914 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:50.628330 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:50.651839 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:50.666904 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:50.868510 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:16:50.926097 19891 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:50.938108 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:51.004371 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:51.022212 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:51.044618 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:51.089958 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:51.100775 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:51.136889 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:51.249301 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:51.384718 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:51.405351 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:51.440472 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:51.452495 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:51.495256 19992 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:51.520442 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:51.532191 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:16:51.540374 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:51.565292 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:51.672422 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:51.675841 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:51.771381 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:51.834820 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:51.843335 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:51.848943 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:51.919639 19891 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:51.930116 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:51.977246 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 14:16:51.987308 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:52.040320 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:52.170537 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:52.174800 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:52.212918 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:52.263381 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 14:16:52.278326 31123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 14:16:52.282371 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:52.303303 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:52.334941 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:52.348726 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:52.361930 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:52.371889 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:52.397017 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 14:16:52.424260 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:16:52.496342 19891 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:52.530124 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 14:16:52.560266 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:52.569591 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:52.599622 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:52.613780 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:52.647071 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:52.688123 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:52.716100 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 14:16:52.736728 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 14:16:52.786589 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:52.855962 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 14:16:52.858505 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:52.899734 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:52.985543 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:52.991530 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:16:53.001022 20114 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:53.031302 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:53.046519 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:53.104498 31034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:16:53.105484 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b1006490ec2841b2a3365295caaf19e1: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:16:53.148576 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:53.201813 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:53.242074 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:53.376508 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:53.426086 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:53.485008 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:53.485585 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:53.500005 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:53.507591 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:53.577598 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:16:53.579069 19891 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:53.635248 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:53.717147 31516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 14:16:53.744515 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:53.771382 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:53.871678 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:53.958896 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:53.996578 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:54.023267 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:54.035034 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:54.077018 20114 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:54.097328 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:54.128788 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:54.150317 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:54.155735 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:54.240226 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:54.252519 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:16:54.347039 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:54.378715 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:54.436991 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:54.462193 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6f8bd03c05d84fdaaf138a6d0608e30e: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:16:54.486730 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:54.502779 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240501 14:16:54.515544 20114 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:54.525835 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:54.611129 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:54.628746 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:54.834656 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:54.841599 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:54.860419 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:54.868719 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 14:16:54.888568 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:54.891283 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:54.892963 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:54.923434 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:54.931031 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:54.956615 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:54.984360 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:16:55.053583 19891 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:55.085917 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:55.118058 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:55.203626 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 14:16:55.239120 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 14:16:55.257284 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:55.289551 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:16:55.298923 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:55.350104 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:55.422693 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:55.436308 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 14:16:55.463608 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:55.474354 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 14:16:55.479216 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:55.496847 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:55.598141 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:55.646337 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:16:55.655571 20310 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:55.667320 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:55.741580 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:55.748420 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:55.808478 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:55.873579 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:55.884954 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:55.912074 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:55.942327 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:56.007289 20327 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:16:56.007424 20327 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:16:56.007905 20327 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:16:56.008262 31188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:16:56.008414 31188 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:16:56.008909 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:56.009023 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:16:56.009099 31518 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:16:56.009224 20327 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:56.024119 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:56.026974 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:16:56.138567 19891 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:56.199667 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:56.206571 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:56.242137 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:56.292562 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:56.320940 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:56.400801 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:56.440815 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:16:56.448280 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:56.502597 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:56.550899 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:16:56.576606 20310 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:56.621119 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:56.672017 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:56.681592 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:56.703624 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:56.733413 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:56.742926 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 14:16:56.789005 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:56.815758 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:56.834165 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:56.960217 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:57.022284 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:57.127395 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:16:57.154719 20310 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:57.244421 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:57.257627 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:57.267155 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:57.293923 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 14:16:57.308316 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:57.312981 31124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 14:16:57.313630 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:57.342257 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 14:16:57.393718 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:57.394359 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:57.403221 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:57.475942 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:57.500474 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:57.502491 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 14:16:57.506912 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:57.520797 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:57.579102 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:57.648422 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 14:16:57.694819 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:57.758977 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:57.762773 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:57.830070 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:57.873713 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:57.930064 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:57.949407 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:58.035382 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:58.051576 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:58.063499 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:58.182641 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:58.202975 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:16:58.208370 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:58.208674 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 14:16:58.234854 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:58.240339 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:58.247176 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 14:16:58.311641 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:58.536327 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:58.581274 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:58.607223 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:58.662114 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:16:58.695535 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:58.723280 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:58.723323 31516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 14:16:58.723945 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:58.724561 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:58.745519 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:58.864341 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:58.886449 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:58.914646 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:58.926308 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:16:59.025362 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 14:16:59.052587 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:16:59.054502 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:59.092790 20468 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:16:59.092952 20468 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:16:59.093479 20468 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:16:59.094535 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:16:59.094647 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:59.094724 31124 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
W20240501 14:16:59.094789 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:16:59.094890 20468 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:16:59.112588 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:16:59.122180 20545 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:59.175063 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:59.229653 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:59.256758 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:59.295209 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:16:59.329259 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:16:59.345808 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:59.352572 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:59.386826 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:16:59.526378 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:16:59.597283 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:16:59.609114 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:16:59.631493 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:59.711881 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:59.799844 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:59.801645 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:16:59.812760 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:16:59.869252 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:16:59.884732 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:16:59.887676 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 14:16:59.951320 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 14:17:00.007457 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:17:00.010821 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:00.077937 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:00.083777 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:17:00.116658 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:00.142320 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:00.212910 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:00.234215 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:17:00.246639 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:17:00.270110 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:17:00.281644 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 14:17:00.290624 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:00.354633 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:00.375793 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:17:00.376353 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:00.507642 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240501 14:17:00.530278 20545 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:00.596590 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:00.614234 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:17:00.672627 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:00.692276 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:00.749590 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:00.752444 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 14:17:00.771412 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:00.825861 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:00.861032 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:01.013636 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:17:01.086448 20545 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:01.106644 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:01.148232 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:01.171664 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:01.191287 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:01.217976 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:01.307633 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:01.320871 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:01.336876 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:01.349556 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:01.380584 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:01.446213 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:01.458026 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:17:01.474273 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 14:17:01.507726 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:01.584805 20545 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:01.602090 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:01.679435 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:01.727025 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:01.752663 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:01.769083 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:01.843958 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:01.879505 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:01.960767 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:01.966362 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:01.976115 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:01.983743 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:02.084434 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:02.099886 20545 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:02.165942 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:02.193375 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:02.272950 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:02.280155 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 14:17:02.318061 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 14:17:02.322443 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:02.322539 31255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 14:17:02.440346 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:02.482863 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:02.505019 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:17:02.534236 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:02.551241 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:02.566530 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 14:17:02.588958 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:02.617218 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:02.652405 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:02.686087 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:02.726460 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:02.745425 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:02.779402 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:02.845594 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 14:17:02.915881 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:02.917840 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:02.980567 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:03.023083 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:03.043706 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:17:03.077482 20865 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:03.084100 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 14:17:03.089749 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 14:17:03.095244 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:03.102322 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:03.126089 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:03.274165 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:03.323851 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:03.469192 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:03.513890 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:03.515866 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:03.540519 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:17:03.553900 20865 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:03.563768 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:03.614506 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:17:03.681265 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:03.700441 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:03.702550 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:03.800534 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:03.844239 31516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 14:17:03.844807 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:03.851045 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:03.896569 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:03.910439 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:03.926681 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:03.958335 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:04.017012 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240501 14:17:04.019187 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:04.021693 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:04.138638 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:04.176793 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:04.212987 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:04.322887 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:04.331506 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:04.398542 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:04.445504 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:04.468163 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:04.528748 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:04.551050 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:04.569993 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:04.578574 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:17:04.586337 20599 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:17:04.586464 20599 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:17:04.587059 20599 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:17:04.588162 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:17:04.588281 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:17:04.588343 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:17:04.588428 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:04.588478 20599 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:04.593847 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:04.594169 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:04.763259 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:04.769467 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:04.796977 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:04.818346 20599 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:17:04.818482 20599 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:17:04.818943 20599 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:17:04.819351 31188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:17:04.819509 31188 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:17:04.819936 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:17:04.820032 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:04.820127 31517 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:17:04.820255 20599 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:04.934118 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:04.997478 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:05.003651 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:05.005942 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 14:17:05.020114 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:05.037372 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:05.042766 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:05.090013 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:05.107302 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:17:05.117309 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:05.161168 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:05.188341 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:05.276628 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:05.278153 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:05.364188 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:05.381379 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:05.382773 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 14:17:05.421567 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 14:17:05.435914 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:17:05.492761 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 14:17:05.496754 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:05.521139 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:05.555202 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:05.556003 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:17:05.624595 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:05.783398 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:17:05.857878 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 14:17:05.877171 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:05.943594 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:05.974041 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:05.980157 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:05.993407 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:06.000075 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:17:06.028188 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:06.182000 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:06.184864 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:06.202733 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:06.237382 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:06.271580 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:06.274391 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 14:17:06.293196 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:06.303095 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:06.383961 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:06.400502 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:06.462064 21054 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:06.483342 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:06.496318 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:06.530112 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:06.569145 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:06.703941 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:06.787215 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:06.839836 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:06.848345 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:06.855942 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:06.861573 21054 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:06.918321 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:06.994087 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:07.005796 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:07.047372 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:07.091984 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:07.147406 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:07.284862 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:07.285806 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:17:07.333160 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:07.343861 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 14:17:07.363425 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:07.368202 31255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240501 14:17:07.381309 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:07.420228 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:07.426813 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:07.460902 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:07.462617 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:07.530051 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:07.546861 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 14:17:07.574733 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:07.623773 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 14:17:07.646003 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:07.662915 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:07.675598 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:07.697024 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:07.724381 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 14:17:07.783111 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:07.792917 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:07.868690 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:07.882912 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:07.893018 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:07.916043 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240501 14:17:07.931741 21054 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:07.979977 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:07.995783 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:08.027643 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:17:08.316900 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 14:17:08.317625 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:08.330461 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:08.362709 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:08.395860 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:08.525550 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:08.526305 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:08.610842 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:08.662685 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:08.691946 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:08.717293 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:08.758086 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 14:17:08.778290 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:08.779492 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:08.783185 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:08.841827 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:08.841967 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:08.866415 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:08.869753 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:08.877229 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:08.877478 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:08.908931 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:08.997920 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:09.006903 31516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 14:17:09.038020 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:09.200929 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:09.225335 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:09.285380 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:09.378686 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:09.422878 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:09.432926 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:17:09.483551 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:09.491139 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:09.501003 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:09.590843 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:09.628576 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:09.712235 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:09.784688 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:09.788727 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:17:09.794415 21005 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:17:09.794562 21005 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:17:09.795073 21005 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:17:09.795554 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:17:09.795722 31187 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:17:09.796121 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:17:09.796311 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:09.796371 31517 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:17:09.796497 21005 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:09.798249 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:09.838315 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:09.899866 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 14:17:09.904654 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:09.958361 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:09.982535 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:09.999177 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:10.006666 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:10.040560 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:10.056159 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:10.063186 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:10.164247 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:10.210322 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 14:17:10.233616 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:10.292763 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:10.301271 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:10.402143 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240501 14:17:10.402326 20468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:10.405408 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:10.413733 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 14:17:10.454907 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 14:17:10.457819 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:10.463289 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:17:10.479607 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:17:10.496493 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:10.500140 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:10.504979 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:10.622025 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:10.890592 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:10.892931 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 14:17:10.902000 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:10.909811 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:10.943522 21448 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:11.065389 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:11.073450 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:11.159000 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:11.177748 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:11.183617 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:11.200856 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:11.216811 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:11.262100 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:11.310945 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:11.318734 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:11.393832 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:11.462179 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:11.490389 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:11.491847 21448 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:11.511947 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:11.526397 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:11.556788 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:11.658078 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:11.715214 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:11.804399 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:11.879745 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:17:11.893153 21494 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:11.944875 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:11.987305 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 14:17:12.035398 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:12.045579 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:12.052029 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:12.094930 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:12.165266 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:12.168751 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:12.257933 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:12.297184 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:12.301307 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:12.309325 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:12.310874 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:12.373148 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 14:17:12.417016 31123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 14:17:12.426695 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:12.447566 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:17:12.488247 21573 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:12.501399 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:12.610774 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:12.619869 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:12.638506 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:12.656116 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:12.666013 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 14:17:12.677645 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 14:17:12.685797 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:12.696260 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:12.853775 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 14:17:12.874545 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:12.884130 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:12.914518 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:12.914646 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:13.076841 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:17:13.086259 21604 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:13.089530 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:13.096210 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:17:13.153669 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:13.195727 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 14:17:13.265591 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:13.386118 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:13.437116 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 14:17:13.515875 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:13.518232 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:13.570629 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:13.588742 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:13.600435 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:13.617528 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:13.656507 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:17:13.684109 21573 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:13.708994 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:13.722241 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:13.768321 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:13.784060 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:13.800737 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:13.901249 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:13.901249 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:14.001325 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:14.067726 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:14.092356 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:14.098510 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 14:17:14.102437 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:14.197026 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:14.235745 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:14.265664 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:17:14.297783 21653 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:14.323503 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:14.324311 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:14.349659 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 14:17:14.400355 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:14.411958 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:14.439880 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:17:14.495337 21653 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:17:14.495510 21653 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:17:14.496170 21653 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:17:14.496729 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:17:14.496909 31581 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:17:14.497278 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:14.497475 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:17:14.497526 31125 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:17:14.497658 21653 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:14.508229 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:14.539148 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:14.558388 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:14.641633 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:14.699527 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:14.715346 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:14.773844 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:14.850116 21653 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:14.862013 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:14.894845 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 14:17:15.002964 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:15.061306 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:15.064498 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:15.129266 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:15.130955 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:15.186986 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 14:17:15.230988 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:15.290100 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:15.307664 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:15.309041 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:15.371225 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:15.431846 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
I20240501 14:17:15.446815 21768 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:15.465564 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:17:15.472975 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:15.497706 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 14:17:15.573351 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:15.630239 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:15.727743 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:15.754418 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:15.816053 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:15.824687 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:17:15.838155 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:15.880924 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:15.952883 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:15.989456 21768 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:16.037501 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:16.061616 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:16.106877 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 14:17:16.109920 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:16.126279 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:16.126993 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:16.170063 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:16.204887 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:16.272248 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:16.284868 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:16.289691 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:16.307096 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:16.309197 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:16.328207 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:16.500432 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:16.546852 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:17:16.559960 21768 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:16.613759 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:16.613811 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:16.624133 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:16.690266 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:16.697232 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:16.816211 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:16.858120 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:16.921000 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:16.930353 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 14:17:16.938145 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:17.020768 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:17.033196 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:17:17.122041 21768 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:17.189448 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:17.254060 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:17.261137 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:17.270570 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:17.302639 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:17.304167 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 14:17:17.318714 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:17.394848 31255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 14:17:17.395527 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:17.424595 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240501 14:17:17.427346 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:17.460386 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:17.507869 21830 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:17.557201 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:17.591595 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:17.690001 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240501 14:17:17.690641 21928 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:17:17.690758 21928 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:17:17.691326 21928 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:17:17.691695 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:17:17.691846 31187 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:17:17.692325 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:17.692399 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:17:17.692483 31518 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:17:17.692638 21928 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:17.730247 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:17.767450 21830 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:17:17.767624 21830 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:17:17.768182 21830 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:17:17.769248 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:17:17.769373 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:17:17.769423 31123 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a0caf2974041408bb32362bfc4284eaa, a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:17:17.769920 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:17.770125 21830 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:17.784427 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:17.803427 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:17.874672 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 14:17:17.877697 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 14:17:17.887859 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:17.893857 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:17.901381 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:17.906363 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:17:17.963752 21928 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:17:17.963896 21928 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:17:17.964356 21928 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:17:17.965256 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:17.965423 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:17:17.965507 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
I20240501 14:17:17.965632 21928 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:17.965631 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:17:18.044319 21991 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:18.124677 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:18.148602 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:18.245621 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:18.265836 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:18.299204 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:18.310590 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:18.333294 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:18.386126 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:17:18.431798 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:18.543480 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 14:17:18.545053 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:18.563138 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:18.568666 21830 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:18.584398 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:18.597882 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:18.622545 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:18.623332 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:18.720038 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:18.838207 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:18.839243 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:18.839732 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:18.874781 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:18.900401 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:18.956851 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:19.024605 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:19.085986 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:19.097259 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:19.103317 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:19.118664 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240501 14:17:19.119268 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:17:19.168160 22076 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:19.219916 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:19.229491 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:19.365993 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 14:17:19.367053 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:19.405067 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:19.444303 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:19.448585 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:19.543207 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:19.583890 22076 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:19.641623 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:19.699944 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:19.733286 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:19.768182 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:19.808015 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:19.821905 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:19.960125 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:19.988104 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:20.002279 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 14:17:20.056605 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:20.075089 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:17:20.141136 22156 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:20.167935 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:20.216539 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:20.232126 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 14:17:20.236757 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:20.256388 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 14:17:20.283584 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:17:20.302521 22044 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:17:20.302661 22044 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:17:20.303128 22044 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:17:20.303450 31188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:17:20.303619 31188 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:17:20.304092 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:20.304198 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:17:20.304257 31518 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:17:20.304387 22044 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:20.414495 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 14:17:20.439656 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:20.508200 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:17:20.508744 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:17:20.602515 22076 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:20.608062 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:20.640398 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:20.646848 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:20.657142 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:20.681145 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:20.708496 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:20.712580 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:20.873080 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:20.880338 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 14:17:20.887141 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:20.950047 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:17:21.003393 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:21.034919 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:21.048790 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:21.136659 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:17:21.190780 22235 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:21.191402 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:21.228080 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:21.235407 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:21.244550 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:21.324316 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:21.324317 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:21.411375 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:21.505903 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:21.544385 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:21.598338 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:21.607602 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:21.622157 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:21.664963 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:21.699560 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:17:21.712961 22235 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:21.723270 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 14:17:21.726162 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:21.798755 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:21.843448 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:21.852631 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:21.880868 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:22.010080 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:22.025346 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:22.079207 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:22.249679 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:22.250463 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:17:22.261969 22235 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:22.275144 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:22.402103 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:22.449568 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240501 14:17:22.450383 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:22.454656 31254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 14:17:22.471930 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:22.517408 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:22.519986 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 14:17:22.551630 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:22.561682 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:22.610008 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:22.611790 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 14:17:22.710740 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:22.815691 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240501 14:17:22.838413 22044 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:17:22.838531 22044 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:17:22.838985 22044 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:17:22.839356 31188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:17:22.839495 31188 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:17:22.839979 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:22.840255 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:17:22.840310 31518 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:17:22.840451 22044 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:17:22.856528 22352 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:22.857559 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:22.916380 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:22.930071 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:22.967052 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 14:17:22.993979 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:23.049453 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:23.106766 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:23.132572 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:23.132650 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:23.277729 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:23.289484 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:23.337127 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:23.351647 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:23.352130 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240501 14:17:23.466343 22351 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:23.471560 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:23.483939 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:17:23.495682 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:23.566556 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:23.568722 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:23.578449 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:23.583686 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:23.694962 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:23.822153 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:23.831660 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:23.876664 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:23.887918 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:17:23.954360 22351 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:23.966643 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:23.982493 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:23.991729 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:24.015885 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:24.021227 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 14:17:24.035019 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:24.127641 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:24.144517 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240501 14:17:24.155679 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:24.157361 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:24.163305 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:24.178020 31032 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 7a542c07a3f841b0a31051a8a37083bf with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 6a6fcc099d1340c383e5a429643d5b7c (127.23.191.67:39413) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: NON_VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6a6fcc099d1340c383e5a429643d5b7c" member_type: VOTER last_known_addr { host: "127.23.191.67" port: 39413 } } peers { permanent_uuid: "30e0b8849e3841ca9ab0de01e719ee4a" member_type: VOTER last_known_addr { host: "127.23.191.66" port: 45265 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
W20240501 14:17:24.213160 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:24.328472 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:24.356707 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:24.371963 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:17:24.379302 22352 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:24.380627 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:24.498412 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:24.502712 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:24.561679 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:24.603186 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:24.766947 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:24.770395 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:17:24.827728 22352 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:24.862735 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:24.884208 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:24.891343 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:24.949775 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:25.012868 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:25.020570 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 14:17:25.050194 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 14:17:25.084494 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 14:17:25.122218 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:25.138193 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:25.329010 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240501 14:17:25.341207 22352 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:25.347133 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:25.383944 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:25.402817 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:25.404142 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:25.418555 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:25.422165 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:25.525813 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:25.541832 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:25.589722 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:25.642668 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 14:17:25.686283 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:17:25.745468 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 14:17:25.774936 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:25.780030 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:17:25.785545 22545 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:25.851625 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:25.897861 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:25.938747 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:25.945513 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:25.946493 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:25.997836 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:26.045595 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:26.057178 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:26.095914 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:26.187575 22352 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:26.367060 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:26.401355 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:26.405725 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 14:17:26.453011 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:26.456167 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:26.468464 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:26.551934 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:26.587455 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:26.589318 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:26.633999 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:17:26.642550 22352 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:26.648350 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:26.672706 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:26.707720 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:26.781808 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:26.833040 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:26.854238 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:26.890518 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:26.892596 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:26.958432 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:26.986529 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:27.003016 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:27.010583 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:27.057933 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:27.063094 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:17:27.098297 22545 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:27.164288 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:27.285574 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:27.381299 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 14:17:27.449404 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 14:17:27.474429 31124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 14:17:27.475082 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:27.479339 22352 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:27.516453 31254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240501 14:17:27.531473 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:27.568943 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:27.604339 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 14:17:27.639166 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:27.758183 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:27.763692 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:27.778343 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:27.797348 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 14:17:27.818163 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:27.897992 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:27.941052 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:17:27.961026 22352 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:27.990176 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:28.009935 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:28.056468 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:28.079371 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:28.101649 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:28.123224 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 14:17:28.162736 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:28.180028 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:28.253129 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:28.259289 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:28.285843 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:17:28.363795 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:28.372160 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:28.456394 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:28.476177 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:28.518976 22779 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:28.618717 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:28.676595 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:28.682947 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:28.727666 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:28.784912 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240501 14:17:28.902734 22779 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:28.924366 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:28.939863 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:28.951089 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:28.997666 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:29.042132 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:29.091358 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:29.134429 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:29.134588 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:29.140671 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:29.169200 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:29.200341 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:29.214702 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 14:17:29.215236 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:29.233924 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:29.291121 22352 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:29.358160 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:29.474345 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:29.518256 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:29.566713 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:29.578480 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:29.626210 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:29.631862 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:29.725800 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:29.738915 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:17:29.785137 22779 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:29.793689 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 14:17:29.810642 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 14:17:29.819401 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:29.842093 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 14:17:29.991958 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:30.052409 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:30.128635 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:17:30.165791 22352 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:30.170811 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:30.184687 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:30.284502 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:30.299317 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:30.312479 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:30.325863 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:30.402664 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:30.409056 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:30.412050 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 14:17:30.420260 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:30.437541 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:30.463778 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 14:17:30.472597 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:30.553169 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:30.592774 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:30.676347 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:17:30.698709 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:30.740994 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:17:30.757632 22949 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:30.762079 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:30.789588 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:30.790148 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:30.806111 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:30.859380 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:17:30.894533 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:31.012365 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:31.140936 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:31.187939 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:17:31.221869 22949 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:31.244422 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:31.271747 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:31.303222 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:31.306010 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 14:17:31.310254 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:31.404110 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:31.414410 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:31.505883 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:31.543442 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:31.547758 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:31.567077 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:31.615557 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:31.713686 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:31.778832 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:31.814489 22949 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:31.817902 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:31.890578 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:31.935762 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:31.938001 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:32.006436 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 14:17:32.034471 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:32.078874 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:32.103103 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:32.142403 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:32.166865 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 14:17:32.188104 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:32.234036 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 14:17:32.246532 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:32.269977 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:32.306135 22949 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:32.402698 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:32.415961 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:32.421914 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:32.482456 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 14:17:32.494135 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:32.537190 31254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 14:17:32.597710 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 14:17:32.639691 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:17:32.681543 22949 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:32.693941 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:32.695883 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:32.795684 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:32.825434 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:32.844691 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:32.898933 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:32.920118 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:32.932482 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:32.978391 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:32.980428 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:32.983238 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 14:17:33.038506 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:17:33.077816 22949 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:17:33.115967 22949 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:17:33.116111 22949 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:17:33.116544 22949 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:17:33.117781 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:17:33.117863 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:33.117924 31124 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
W20240501 14:17:33.118039 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:17:33.118072 22949 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:33.141749 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:17:33.172570 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:33.194841 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:33.291985 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:33.293087 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:33.328279 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:33.432356 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:33.455483 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:33.456576 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:33.527169 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:17:33.556615 22949 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:33.611537 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:33.649168 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:33.720012 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:33.747475 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 14:17:33.773674 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:33.793004 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:33.799304 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:33.860651 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:33.861845 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:34.024652 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:34.052584 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:34.065215 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:34.081833 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:34.134078 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:34.139189 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:17:34.166152 22949 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:34.196787 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:34.227710 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 14:17:34.228345 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:34.287683 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:34.296773 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:34.369053 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:34.391311 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:34.443729 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:34.504299 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 14:17:34.553082 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:17:34.596894 23027 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:34.639065 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:34.670228 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:34.726006 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:34.755431 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:34.808413 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 14:17:34.818325 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:34.877589 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:34.893622 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 14:17:34.927145 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:34.988261 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:35.043676 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:35.064291 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 14:17:35.093771 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:35.108258 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:35.129703 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:35.134065 23265 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:35.231458 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:35.278182 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:35.348150 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:35.352381 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:35.388005 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:35.438928 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:35.522375 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 14:17:35.544242 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:35.547421 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:35.620034 23027 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:35.641861 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:35.665354 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:35.686811 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:17:35.767583 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:35.790073 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:35.805784 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:35.825484 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:35.884966 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:35.913161 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:35.944799 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:17:35.964444 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:17:36.026065 23265 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:36.066655 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:36.087810 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:36.116169 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:36.144513 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:36.147492 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:36.149112 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 14:17:36.187971 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:36.277379 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:36.313269 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:36.383494 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:17:36.536028 23027 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:36.536800 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:36.545295 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:36.551846 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:36.613684 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:36.620988 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:36.650620 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:36.721168 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 14:17:36.736045 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:36.761837 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:36.858736 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:36.929389 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:36.948498 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:37.076511 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:37.088651 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:37.133713 23027 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:37.134807 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:37.145174 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:37.187641 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:37.224417 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:37.348325 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:37.394750 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 14:17:37.407321 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:37.414935 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:37.453251 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 14:17:37.488232 31124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240501 14:17:37.488909 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:37.497651 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240501 14:17:37.511145 23027 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:37.533989 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:37.590024 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:37.602254 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 124 similar messages]
I20240501 14:17:37.626631 23390 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:17:37.626762 23390 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:17:37.627243 23390 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:17:37.628473 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:17:37.628569 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:37.628657 31517 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
W20240501 14:17:37.628736 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:17:37.628793 23390 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:37.649293 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:37.680492 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:37.704813 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:37.774689 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:37.781149 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:37.843370 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:37.852965 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:37.855613 23027 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:17:37.855738 23027 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:17:37.856295 23027 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:17:37.856909 31581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:17:37.857060 31581 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:17:37.857492 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:17:37.858114 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:37.858177 31124 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:17:37.858319 23027 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:37.894343 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:37.917001 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:17:38.053833 23373 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:38.098918 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:38.124989 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 14:17:38.143927 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:38.182844 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:38.260226 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:38.261883 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:38.279661 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:38.289767 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:38.317139 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:38.395095 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:17:38.403896 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 14:17:38.405844 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:17:38.491549 23027 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:38.503427 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:38.515116 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:38.622321 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:38.629209 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:38.670985 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:38.675985 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:38.742779 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:38.764714 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:38.781610 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:38.785327 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:38.864967 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:38.947227 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:17:38.965066 23027 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:39.009335 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:39.020139 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:39.049433 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:39.093111 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:39.190378 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:39.230224 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 14:17:39.324573 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:39.349737 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:39.371070 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:39.412711 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:17:39.562129 23027 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:39.566505 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:39.601397 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 14:17:39.730691 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:39.742923 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 14:17:39.784375 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:39.811007 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:39.887032 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:39.943379 23599 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:39.947414 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:39.985987 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:39.993363 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 14:17:40.005209 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:40.005925 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:40.025626 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:40.030709 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:40.031173 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:40.055658 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 14:17:40.191788 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:40.220167 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:40.228011 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:40.232056 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:40.234642 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:40.310364 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:40.341961 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:40.370641 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:40.417888 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:17:40.536337 23599 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:40.552232 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:40.565701 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:40.617547 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:40.717371 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:17:40.717902 23600 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:17:40.718014 23600 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:17:40.718472 23600 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:17:40.718806 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:17:40.718953 31187 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:17:40.719272 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:40.719424 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:17:40.719507 31518 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:17:40.719633 23600 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:40.744887 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 14:17:40.756060 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:17:40.825703 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:40.825837 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:40.829947 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:40.842376 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:40.887578 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:40.918818 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 14:17:41.033165 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240501 14:17:41.088613 23599 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:41.114450 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:41.142755 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:41.181896 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:41.199712 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:41.222188 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:41.241978 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:41.273571 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:41.354240 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:41.410678 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:41.437002 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:41.500432 23027 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:41.556167 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:41.588447 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:41.651686 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:41.702657 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:41.736773 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:41.774259 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:41.821326 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:41.834805 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:41.884268 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:41.890559 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:41.968959 23599 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:42.012493 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 14:17:42.037544 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:42.139645 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:42.199137 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 14:17:42.303082 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:42.319132 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:42.342895 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:42.393127 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:42.395730 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 14:17:42.465848 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:17:42.523053 23678 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:17:42.523216 23678 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:17:42.523684 23678 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:17:42.524907 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:17:42.525043 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:17:42.525072 23599 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:17:42.525102 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:17:42.525264 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:42.525249 23678 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:42.554219 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:42.565660 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240501 14:17:42.566591 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:42.581389 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:42.581423 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:42.584908 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:42.605587 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:42.619766 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240501 14:17:42.661523 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 14:17:42.700795 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:42.743973 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:42.759114 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:42.759142 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:42.802058 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:42.819528 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:42.954522 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:42.959761 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:43.001093 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:43.033627 23818 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:43.035358 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:17:43.036749 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:43.160467 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 14:17:43.281286 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:43.292981 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:43.381126 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:43.401360 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:43.425204 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:43.460309 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:43.481168 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:17:43.504598 23599 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:43.517509 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:43.545596 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:43.582074 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:43.606133 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:43.650029 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:43.693542 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:43.722084 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:43.744246 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:43.746718 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 14:17:43.866927 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:43.928989 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:17:43.955816 23599 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:43.970395 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:43.983184 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:44.010805 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:44.111802 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:44.130319 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:44.142268 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:44.279728 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240501 14:17:44.281359 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:44.322510 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:44.373389 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
I20240501 14:17:44.376438 23818 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:44.434458 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:44.465699 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:44.471871 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:44.488762 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:44.505824 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 14:17:44.520655 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:44.667109 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:44.713384 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:44.849977 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:17:44.864729 23818 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:44.912760 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 14:17:44.922125 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:44.960606 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:44.988480 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:45.018433 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:45.018944 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:45.037446 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:45.038303 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:45.047670 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:45.121762 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 14:17:45.154595 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:45.195276 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:45.195495 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:45.250468 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:45.393183 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:45.411532 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:17:45.436312 23818 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:45.456450 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:45.508730 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:45.536214 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:45.577121 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:45.722056 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:45.751765 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:17:45.838286 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 14:17:45.861197 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:45.878544 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:45.878624 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:17:45.933503 23599 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:45.970180 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:17:45.981459 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:46.001281 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:46.050120 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:46.109419 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:46.115082 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:46.129858 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:46.144126 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:46.199060 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:46.267697 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 14:17:46.271749 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:46.293556 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:46.296115 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:46.355484 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:46.357302 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:46.424425 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:46.516770 24113 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:46.573144 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:46.580272 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:46.588672 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:46.607273 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:46.645141 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:46.714556 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:46.740923 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 14:17:46.757290 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:46.827945 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:46.874200 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 14:17:46.878659 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:17:46.960484 24113 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:46.995012 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:47.156412 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:47.238628 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:47.242302 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:47.313161 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:47.333909 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:47.345767 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:47.353826 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:47.374285 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:47.380815 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 14:17:47.440600 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 14:17:47.442310 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:47.526262 24113 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:47.536388 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:47.545533 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:47.599071 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:47.638592 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:47.656819 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:47.662372 31123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 14:17:47.695170 31256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 14:17:47.722899 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:47.739178 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:47.813738 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:47.844817 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:47.895247 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:47.901324 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:47.909819 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:47.914431 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:17:47.926687 24113 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:48.100955 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:17:48.117795 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:48.252310 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:48.271145 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
I20240501 14:17:48.321264 24113 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:17:48.321391 24113 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:17:48.321813 24113 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:17:48.322839 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:48.322922 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:17:48.322978 31125 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a0caf2974041408bb32362bfc4284eaa, a63e0dd1ba2e4a8db8ace9718a700c32
W20240501 14:17:48.323067 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:48.323124 24113 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:48.330533 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:48.345077 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:48.362656 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:48.377486 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:48.383044 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:48.389117 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:17:48.417022 24113 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:48.430794 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:48.600998 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:48.628602 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:48.648463 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:48.665874 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:48.675985 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:48.741937 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:17:48.788137 24113 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:17:48.788287 24113 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:17:48.788727 24113 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7eada5032fb45c3a5e3196e2b3faaab (127.23.191.68:45349), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:17:48.789149 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "0e4622d80cc94f12b492bdd7597a61e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" is_pre_election: true
I20240501 14:17:48.789305 31580 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e4622d80cc94f12b492bdd7597a61e1 in term 2.
W20240501 14:17:48.789768 31125 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:17:48.790228 31124 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:48.790302 31124 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:17:48.790436 24113 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:48.881899 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:48.883428 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:48.890686 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:48.911799 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:48.915800 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:48.945202 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:48.951452 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:48.973191 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
I20240501 14:17:49.019063 24113 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:49.046113 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:49.150812 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:49.231376 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:49.258893 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 14:17:49.286550 31516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 14:17:49.287184 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:49.374078 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 14:17:49.387787 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:49.392556 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:49.466650 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:17:49.515836 24113 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:49.540437 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:49.603857 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:49.755970 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:49.772441 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:49.776878 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 14:17:49.811620 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:49.867955 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:49.910907 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:49.919193 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:49.923475 24113 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:49.970324 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:49.989079 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:50.010457 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:50.135870 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:50.147498 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:50.193809 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:50.221222 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:50.278887 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:50.317176 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 14:17:50.318502 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:50.333272 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:50.342854 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:50.347712 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:50.389286 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:50.462896 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:17:50.504925 23678 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:17:50.505059 23678 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:17:50.505534 23678 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:17:50.506489 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:50.506592 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:17:50.506678 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
I20240501 14:17:50.506816 23678 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:17:50.506824 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
I20240501 14:17:50.537232 24113 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:50.545966 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:50.720937 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:50.733981 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:50.740862 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 14:17:50.771942 31387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 14:17:50.787228 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:50.804401 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:50.853386 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:50.896239 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:17:50.921504 24306 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:50.934245 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:17:50.959599 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:51.103914 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:51.118536 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:51.182366 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:51.185287 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:51.206539 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:51.233225 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:51.258496 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:51.284803 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:51.286073 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:51.320919 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:51.337019 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:51.450716 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:51.463835 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:17:51.526625 24498 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:51.529091 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 14:17:51.561702 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:51.575570 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:51.692759 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:51.693347 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:51.696329 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 14:17:51.764770 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:51.914000 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 14:17:51.994657 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:51.995849 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:17:52.009505 24306 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:52.043673 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:52.112069 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:52.131274 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:52.216734 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:52.260587 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:52.271091 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:52.272455 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:52.333055 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:52.410857 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:17:52.425966 24306 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:52.426483 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:52.462165 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:52.475410 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:52.491346 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 14:17:52.612323 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:52.631395 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:52.641638 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:52.650841 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:52.697494 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240501 14:17:52.698027 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 14:17:52.705075 31255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 14:17:52.768690 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:52.770525 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:52.777565 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:52.805604 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:52.848047 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:17:52.998339 24306 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:53.027369 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:53.046778 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:53.052469 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:53.062654 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:53.131464 31034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:17:53.132189 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b83f4cd48b4b4251a8ac4ddebdb8e4e1: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:17:53.166523 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:53.224620 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:53.292520 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:53.301988 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:53.355885 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:53.440411 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:53.446496 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:17:53.492048 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:53.505134 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:53.538224 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
I20240501 14:17:53.543074 24306 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:53.566025 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:53.589912 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:53.749810 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:53.782310 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:53.785156 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:53.946413 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 14:17:53.963457 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:53.970943 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:53.977452 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:54.011999 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:54.018855 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:54.073640 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:54.092218 24687 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:54.103106 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:54.161757 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:54.204555 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:54.327704 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 14:17:54.327977 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:54.354178 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 14:17:54.383565 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:54.428937 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:54.479523 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:54.488014 31034 catalog_manager.cc:4573] TS a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6f8bd03c05d84fdaaf138a6d0608e30e: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:17:54.510679 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
I20240501 14:17:54.560662 24645 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:54.596619 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:54.621134 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:54.651790 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:54.671355 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:54.688400 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:54.776131 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:54.857373 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:54.881114 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:54.970235 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:17:54.981989 24645 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:55.019491 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:55.036338 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:55.126657 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:55.128780 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:55.129693 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 14:17:55.163241 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:55.170161 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:55.191169 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:55.219869 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:55.256701 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:55.277930 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 14:17:55.337706 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:55.349920 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:17:55.369504 24645 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:55.377678 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:55.385051 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:55.483691 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:55.576704 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:55.585939 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:55.603857 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:17:55.639448 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:55.681879 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:55.768563 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:55.784904 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:55.818743 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 76 similar messages]
I20240501 14:17:55.904070 24645 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:55.947371 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:55.986220 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:56.018170 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:56.046218 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:56.097970 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 14:17:56.190188 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:56.214031 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:56.229400 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:56.245661 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:56.280345 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:17:56.325062 24645 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:56.373595 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 14:17:56.428961 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:56.472386 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:56.493677 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:56.498538 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:56.603686 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:56.605114 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:56.629598 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:56.785012 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:56.827472 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:56.835999 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:17:56.839066 24820 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:56.846189 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:56.878304 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:56.896854 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:56.956200 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 14:17:56.968753 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 14:17:56.982292 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:57.040894 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:57.092640 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:57.092995 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:57.196437 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:57.232430 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:57.313987 24820 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:57.397794 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:57.438037 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:57.447093 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:57.466295 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:57.467896 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:57.601255 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:57.715870 31254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 14:17:57.736532 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:57.737170 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:17:57.737874 24820 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:57.786249 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:17:57.788998 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:57.808483 31125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240501 14:17:57.835492 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:57.841171 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:57.849871 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 14:17:57.850026 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:57.884124 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:57.889756 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 14:17:57.917330 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:57.950417 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:58.019536 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:17:58.033434 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:58.181325 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:17:58.233381 24820 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:58.255749 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:58.282104 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:58.338745 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:58.355794 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 14:17:58.361266 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:58.441150 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:17:58.461153 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:17:58.571283 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:58.577718 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:58.679562 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:17:58.737552 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
I20240501 14:17:58.808449 24645 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:58.819000 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:58.858170 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:58.872306 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:58.881448 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:58.896109 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:58.933768 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:58.946646 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:58.961094 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:58.970212 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:17:59.029870 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:59.255699 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:59.329385 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:59.356190 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:17:59.370335 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
I20240501 14:17:59.387806 25079 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:59.404959 31518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 14:17:59.420661 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:17:59.443039 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:59.447743 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:59.496580 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:17:59.531487 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:17:59.532349 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:59.566376 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:17:59.583863 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 14:17:59.607517 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:59.616051 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:17:59.769079 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:17:59.811110 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:17:59.861872 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:59.938712 25116 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:17:59.940570 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:59.962419 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:17:59.962910 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:17:59.971313 25116 raft_consensus.cc:482] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:17:59.971429 25116 raft_consensus.cc:504] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "a0caf2974041408bb32362bfc4284eaa" member_type: VOTER last_known_addr { host: "127.23.191.71" port: 35407 } attrs { promote: false } }
I20240501 14:17:59.972007 25116 leader_election.cc:288] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407)
W20240501 14:17:59.972867 31125 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
W20240501 14:17:59.972957 31124 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:17:59.973033 31124 leader_election.cc:302] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
W20240501 14:17:59.973136 31123 leader_election.cc:334] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:17:59.973155 25116 raft_consensus.cc:2726] T 5b2e30cb9de14fe19273b3cef801cb67 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:18:00.063573 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:18:00.067639 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:00.132582 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:18:00.182917 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:00.225522 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:00.255586 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 14:18:00.407023 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:00.416406 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:00.438688 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:18:00.472066 25116 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:00.489777 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 14:18:00.513522 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:00.582214 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:00.587266 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:18:00.590283 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:18:00.623498 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:18:00.728667 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:00.733268 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:00.782155 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:00.816582 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:00.820689 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:18:00.921126 25160 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:00.927671 31386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:18:01.000339 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:01.013826 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:01.029134 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 14:18:01.067721 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:01.118386 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:01.183218 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:01.183358 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:01.199630 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 14:18:01.305356 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:01.354205 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:18:01.356986 25116 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:01.361595 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:01.382509 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:01.399693 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:01.404633 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:01.444341 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:01.468585 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:01.583577 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:01.746412 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:01.749550 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:01.790130 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:01.800395 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:01.806082 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:18:01.828048 25116 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:01.881162 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:01.927647 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 14:18:01.940168 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 14:18:01.945142 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:01.994272 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:02.144574 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:02.148003 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:02.156976 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:02.185675 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:02.213912 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:02.220291 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:02.288007 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:02.382834 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:02.423807 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:18:02.429308 25310 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:02.472115 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:02.609578 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:02.637038 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:02.659835 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 14:18:02.678782 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:02.684762 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:02.735981 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:02.755249 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:02.771847 31254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 14:18:02.813781 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:18:02.842849 24752 raft_consensus.cc:482] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b934fd09590a40ada094b9715d990a00)
I20240501 14:18:02.842969 24752 raft_consensus.cc:504] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0e4622d80cc94f12b492bdd7597a61e1" member_type: VOTER last_known_addr { host: "127.23.191.65" port: 32841 } attrs { replace: true } } peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:18:02.843418 24752 leader_election.cc:288] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e4622d80cc94f12b492bdd7597a61e1 (127.23.191.65:32841), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
I20240501 14:18:02.843776 31187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f5e4e25a674a99a6762757f0ab1545" candidate_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "0e4622d80cc94f12b492bdd7597a61e1" is_pre_election: true
I20240501 14:18:02.843938 31187 raft_consensus.cc:2454] T 96f5e4e25a674a99a6762757f0ab1545 P 0e4622d80cc94f12b492bdd7597a61e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7eada5032fb45c3a5e3196e2b3faaab in term 2.
W20240501 14:18:02.844455 31518 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:18:02.844588 31517 leader_election.cc:334] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:18:02.844645 31517 leader_election.cc:302] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0e4622d80cc94f12b492bdd7597a61e1, e7eada5032fb45c3a5e3196e2b3faaab; no voters: 9438789dc61f4501ad1d4ceaeccee458, b934fd09590a40ada094b9715d990a00
I20240501 14:18:02.844769 24752 raft_consensus.cc:2726] T 96f5e4e25a674a99a6762757f0ab1545 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:18:02.866940 31124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:18:02.867499 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:02.924710 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:18:03.018326 25310 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:03.028820 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:03.102424 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:03.128139 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:03.133967 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 14:18:03.159685 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:03.171227 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:03.245504 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:18:03.272089 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:03.346792 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:03.362071 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:03.372787 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:03.417687 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:03.434482 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
I20240501 14:18:03.515609 25310 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:03.568186 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:03.569464 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:03.579066 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:03.613271 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:03.801498 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:03.828565 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:03.835055 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:03.860770 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 14:18:03.871248 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:03.932350 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:03.995579 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:18:04.071830 25310 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:04.088158 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:04.105243 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:04.221868 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:04.301792 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:04.303077 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:04.306905 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:04.410399 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:04.450731 31516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 14:18:04.451311 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:04.459579 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:04.534699 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:04.571314 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 14:18:04.577680 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:18:04.584290 25310 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:04.592022 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:04.636348 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240501 14:18:04.659574 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:04.721225 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:04.763692 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:04.803061 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:04.803220 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:04.935163 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:04.958369 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:18:05.004102 25468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:05.052086 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:05.067821 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:05.089586 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:05.127264 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:05.168622 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:05.217969 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:05.250073 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 14:18:05.286062 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:05.311820 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:05.312402 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:05.338639 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:18:05.405107 25310 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:05.415243 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:05.458917 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:05.542025 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:05.546239 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:05.578480 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:05.642839 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:05.681818 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:05.720896 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:18:05.766003 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:05.768585 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:05.813211 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:05.826774 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:05.851799 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:05.854348 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 14:18:05.894479 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 14:18:05.913554 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:18:06.001453 25468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:06.013206 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:18:06.013815 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:06.075048 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:06.256342 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:06.308521 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:06.314571 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:06.325603 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:06.354900 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:06.379173 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:18:06.459812 25310 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:06.578033 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:06.604462 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 14:18:06.604873 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:06.607786 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:06.731642 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:06.741091 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:06.746681 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:06.759586 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:06.780869 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:06.809571 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
I20240501 14:18:06.866139 25468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:06.935237 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240501 14:18:06.986763 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:06.996062 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:07.034821 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:07.070122 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:07.135746 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 14:18:07.159359 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:07.268096 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:18:07.270804 25468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:07.282565 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:07.299775 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:07.334622 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:07.354804 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:07.383352 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:07.423267 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:07.469599 31254 consensus_peers.cc:485] T f8cc4d3985224178aa008b4c9de5625b P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:18:07.514530 31255 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:07.656980 31517 consensus_peers.cc:485] T 9d2c173b2c32494eb16e46a51ca36e31 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:07.670238 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:07.688038 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:07.692936 31254 consensus_peers.cc:485] T b67bd0eb45a3486492b75ca3ad3be524 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:07.721454 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:07.725661 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 14:18:07.738847 31517 consensus_peers.cc:485] T d265eff200b645bfac8f8906654385f6 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:18:07.761731 25416 raft_consensus.cc:482] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:18:07.761893 25416 raft_consensus.cc:504] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7eada5032fb45c3a5e3196e2b3faaab" member_type: VOTER last_known_addr { host: "127.23.191.68" port: 45349 } attrs { replace: true } } peers { permanent_uuid: "a63e0dd1ba2e4a8db8ace9718a700c32" member_type: VOTER last_known_addr { host: "127.23.191.72" port: 39799 } attrs { promote: false } } peers { permanent_uuid: "b934fd09590a40ada094b9715d990a00" member_type: VOTER last_known_addr { host: "127.23.191.69" port: 45429 } attrs { promote: false } } peers { permanent_uuid: "9438789dc61f4501ad1d4ceaeccee458" member_type: VOTER last_known_addr { host: "127.23.191.70" port: 38181 } attrs { promote: false } }
I20240501 14:18:07.762421 25416 leader_election.cc:288] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799), b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429), 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181)
W20240501 14:18:07.763518 31517 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:07.763727 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111)
I20240501 14:18:07.763795 31518 leader_election.cc:302] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e7eada5032fb45c3a5e3196e2b3faaab; no voters: a63e0dd1ba2e4a8db8ace9718a700c32, b934fd09590a40ada094b9715d990a00
I20240501 14:18:07.763932 25416 raft_consensus.cc:2726] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:18:07.763952 31518 leader_election.cc:334] T c082ccc777ba4f9a937931f707645e54 P e7eada5032fb45c3a5e3196e2b3faaab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111)
W20240501 14:18:07.801386 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:07.832849 31255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 118 similar messages]
I20240501 14:18:07.833077 25468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:07.842551 31387 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:07.869552 31124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 14:18:07.870102 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:07.923969 31254 consensus_peers.cc:485] T 5d994081c3414f779ba2cae431cb1d97 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:07.945845 31123 consensus_peers.cc:485] T 71241540f585474a8f62da16ed2d9874 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:18:08.071065 31385 consensus_peers.cc:485] T 7a542c07a3f841b0a31051a8a37083bf P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:08.086879 31518 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:08.090107 31125 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:08.135962 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:08.181392 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:08.182785 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 14:18:08.211508 31124 consensus_peers.cc:485] T 28013d7df97943e09c27c3bb1794ee80 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:08.223743 31387 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:18:08.223774 31516 consensus_peers.cc:485] T 72ad9c714b2b4fff94063d2838835173 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:08.256597 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:08.323062 31125 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 14:18:08.344692 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
I20240501 14:18:08.379099 25468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:08.416041 31125 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:08.558020 31517 consensus_peers.cc:485] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:18:08.559087 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:08.657132 31385 consensus_peers.cc:485] T ea66fe3d0da04d18b961927a9c988a20 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:18:08.749217 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:08.780685 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:08.849615 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:08.870466 31254 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:08.888278 31125 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:18:08.963027 25798 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:09.002719 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:09.103066 31386 consensus_peers.cc:485] T 8ffdb76b2c5149a18d45bee9581e605a P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:09.113006 31256 consensus_peers.cc:485] T 8f91909d3b1b46e9b858ce5c5b2e7975 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240501 14:18:09.136766 31385 consensus_peers.cc:485] T ad0dac76fd0c4ae6b8e0549376c98e5d P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:18:09.161645 31255 consensus_peers.cc:485] T 3c8200d9c20e4140906c71c746badb94 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:18:09.162940 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer 9438789dc61f4501ad1d4ceaeccee458 (127.23.191.70:38181): Couldn't send request to peer 9438789dc61f4501ad1d4ceaeccee458. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.70:38181: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240501 14:18:09.220309 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:09.239202 31123 consensus_peers.cc:485] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:09.322296 31125 consensus_peers.cc:485] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:18:09.332965 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:09.340528 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:09.372859 31516 consensus_peers.cc:485] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:18:09.413671 25798 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:09.426332 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:09.481971 31517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240501 14:18:09.491307 31256 consensus_peers.cc:485] T 0399a419a25c4981a2ee90d76cbf59b1 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer a63e0dd1ba2e4a8db8ace9718a700c32 (127.23.191.72:39799): Couldn't send request to peer a63e0dd1ba2e4a8db8ace9718a700c32. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.72:39799: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240501 14:18:09.598275 31517 consensus_peers.cc:462] T 2e68be1521964ba8b864801fdf3b0df3 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:09.650359 31255 consensus_peers.cc:485] T 4603d1b1b1d441b1b0be0464e2f2c9e2 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:09.691656 31124 consensus_peers.cc:485] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:18:09.730010 31124 consensus_peers.cc:462] T c0b7ede9ffaf496e9b106b5d05ed25a5 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:09.778708 31385 consensus_peers.cc:485] T 043344ce36a74a18a0d67ff5bd473e68 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn't send request to peer a0caf2974041408bb32362bfc4284eaa. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:18:09.809005 25468 raft_consensus.cc:1013] T a7810938b8fb406e9c6ac438f54afd03 P 0e4622d80cc94f12b492bdd7597a61e1 [term 1 LEADER]: attempt to promote peer e7eada5032fb45c3a5e3196e2b3faaab: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 14:18:09.865711 31516 consensus_peers.cc:462] T b83f4cd48b4b4251a8ac4ddebdb8e4e1 P e7eada5032fb45c3a5e3196e2b3faaab -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.71:35407: connect: Connection refused (error 111)
W20240501 14:18:09.898970 31124 consensus_peers.cc:462] T bca09920a7904362bc546916e3f6f5ac P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111)
W20240501 14:18:09.970177 31255 consensus_peers.cc:485] T f26d16499ab54a3e9059f6e4ff041177 P 30e0b8849e3841ca9ab0de01e719ee4a -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:10.053768 31517 consensus_peers.cc:485] T 6bcd6a2f3c014d5ca0e32e16cdbc4edc P e7eada5032fb45c3a5e3196e2b3faaab -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:18:10.120256 31386 consensus_peers.cc:485] T 5aa7bdac7bd1408cbf41d894d40aebd4 P 6a6fcc099d1340c383e5a429643d5b7c -> Peer b934fd09590a40ada094b9715d990a00 (127.23.191.69:45429): Couldn't send request to peer b934fd09590a40ada094b9715d990a00. Status: Network error: Client connection negotiation failed: client connection to 127.23.191.69:45429: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:18:10.133550 31123 consensus_peers.cc:485] T b1006490ec2841b2a3365295caaf19e1 P 0e4622d80cc94f12b492bdd7597a61e1 -> Peer a0caf2974041408bb32362bfc4284eaa (127.23.191.71:35407): Couldn'